Jul 09 18:02:37.198411 ip-10-0-140-229 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 18:02:37.198422 ip-10-0-140-229 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 18:02:37.198429 ip-10-0-140-229 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 18:02:37.198670 ip-10-0-140-229 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 18:02:47.424741 ip-10-0-140-229 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 18:02:47.424764 ip-10-0-140-229 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2551a8a5ad0c4f8eb53543c251f47a85 --
Jul 09 18:04:37.388901 ip-10-0-140-229 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:04:37.912115 ip-10-0-140-229 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:37.912115 ip-10-0-140-229 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:04:37.912115 ip-10-0-140-229 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:37.912115 ip-10-0-140-229 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:04:37.912115 ip-10-0-140-229 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:37.914250 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.914170    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:04:37.918354 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918336    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.918354 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918352    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.918354 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918357    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918362    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918367    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918372    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918377    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918381    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918385    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918389    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918393    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918397    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918402    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918406    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918410    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918414    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918418    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918423    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918427    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918431    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918435    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918439    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918442    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.918542 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918446    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918450    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918454    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918458    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918461    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918465    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918470    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918474    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918478    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918482    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918486    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918490    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918495    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918499    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918503    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918515    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918522    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918529    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918535    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.919421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918540    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918544    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918549    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918553    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918558    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918562    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918566    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918571    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918575    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918579    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918583    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918588    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918592    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918596    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918601    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918605    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918624    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918629    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918633    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918637    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.920168 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918642    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918646    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918650    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918654    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918658    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918662    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918667    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918672    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918676    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918682    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918686    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918690    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918695    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918699    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918706    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918712    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918717    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918722    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918726    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918730    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.920896 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918734    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918739    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918743    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918747    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918753    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918757    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918762    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918766    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918770    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918775    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918779    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918783    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918788    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918791    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918796    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918800    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.921489 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.918805    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.921972    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.921984    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.921989    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.921992    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.921995    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.921998    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922001    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922004    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922007    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922009    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922012    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922015    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922018    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922020    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922023    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922025    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922028    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922030    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922033    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922036    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.922039 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922038    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922041    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922043    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922046    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922049    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922053    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922058    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922061    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922064    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922067    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922069    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922073    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922075    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922078    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922080    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922083    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922085    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922087    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922090    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.922574 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922092    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922095    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922097    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922099    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922102    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922104    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922107    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922110    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922113    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922115    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922118    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922120    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922123    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922125    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922128    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922131    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922134    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922136    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922139    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922141    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.923067 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922144    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922147    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922149    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922152    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922154    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922157    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922159    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922162    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922164    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922166    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922169    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922172    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922174    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922176    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922180    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922183    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922186    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922189    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922191    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.923537 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922194    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922196    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922199    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922201    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922205    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922209    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922213    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922215    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922219    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922223    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922225    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922228    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922230    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922233    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922235    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922238    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922240    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922243    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922246    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.924006 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922248    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922250    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922641    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922648    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922651    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922656    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922659    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922663    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922666    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922669    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922672    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922675    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922678    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922681    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922684    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922687    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922690    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922692    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922696    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922699    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.924447 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922702    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922705    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922708    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922711    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922713    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922716    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922718    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922721    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922723    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922726    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922728    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922731    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922733    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922736    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922738    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922740    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922743    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922746    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922748    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922750    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.924940 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922754    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922758    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922761    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922763    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922766    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922768    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922771    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922773    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922776    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922778    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922781    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922785    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922787    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922790    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922792    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922795    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922797    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922800    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922802    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922805    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.925414 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922807    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922809    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922812    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922814    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922817    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922820    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922822    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922824    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922848    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922851    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922855    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922859    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922862    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922865    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922868    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922871    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922873    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922876    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922879    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.925897 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922881    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922884    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922886    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922889    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922891    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922896    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922899    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922902    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922904    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922906    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922909    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922911    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922914    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922917    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922920    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922922    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922925    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922927    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922930    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922933    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.926345 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922936    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.922939    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923000    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923015    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923023    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923029    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923035    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923040    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923048    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923052    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923056    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923060    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923063    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923066    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923069    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923073    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923076    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923079    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923082    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923085    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923090    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923093    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923096    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:04:37.926852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923099    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923103    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923111    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923114    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923117    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923120    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923123    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923126    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923130    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923133    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923135    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923140    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923143    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923145    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923148    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923151    2561 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923154    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923158    2561 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923161    2561 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923164    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923167    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923169    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923173    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923176    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923179    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:04:37.927403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923181    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923184    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923187    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923190    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923193    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923196    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923199    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923201    2561 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923205    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923208    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923211    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923214    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923216    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923219    2561 flags.go:64] FLAG: --help="false"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923222    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-140-229.ec2.internal"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923225    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923228    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923231    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923234    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923238    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923241    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923244    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923247    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:04:37.928035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923250    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923253    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923256    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923259    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923261    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923264    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923267    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923270    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923273    2561 flags.go:64] FLAG: --lock-file=""
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923275    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923278    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923281    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923286    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923289    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923292    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923295    2561 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923297    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923301    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923303    2561 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923306    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923310    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923313    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923317    2561 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923320    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923323    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:04:37.928592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923325    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923328    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923331    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923334    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923337    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923343    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923349    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923352    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923355    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923358    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923363    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923366    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923369    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923372    2561 flags.go:64] FLAG: --port="10250"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923375    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923378    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d1fa53ceb9b0e77e"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923381    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923384    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923387    2561 flags.go:64] FLAG: --register-node="true"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923389    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923393    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923396    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923399    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923401    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923405    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:04:37.929213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923408    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923411    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923413    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923416    2561 flags.go:64] FLAG: --runonce="false"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923419    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923422    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923425    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923428    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923430    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923433    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923437    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923439    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923442    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923447    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923450    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923453    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923457    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923459    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923462    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923467    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923470    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923473    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923477    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923480    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923483    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:04:37.929839 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923485    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923488    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923491    2561 flags.go:64] FLAG: --v="2"
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923495    2561 flags.go:64] FLAG: --version="false"
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923499    2561 flags.go:64] FLAG: --vmodule=""
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923503    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.923506    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923630    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923636    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923640    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923643    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923647    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923649    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923652    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923656    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923659    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923662    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923665    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923667    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923670    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923673    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.930428 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923676    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923679    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923682    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923684    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923687    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923690    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923692    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923694    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923697    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923700    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923702    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923704    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923707    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923709    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923712    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923715    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923718    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923720    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923723    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.930937 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923726    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923728    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923731    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923733    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923736    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923738    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923742    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923745    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923748    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923750    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923753    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923755    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923759    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923762    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923765    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923767    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923770    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923772    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923775    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923777    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.931421 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923780    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923782    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923785    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923788    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923790    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923793    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923795    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923798    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923801    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923803    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923807    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923809    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923812    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923814    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923817    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923819    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923821    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923824    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923828    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923830    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.931922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923833    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923835    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923838    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923840    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923844    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923847    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923849    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923852    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923855    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923857    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923860    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923862    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923865    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923868    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923872    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923877    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923880    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923883    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923886    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923889    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.932431 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923891    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923894    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923897    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923901    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923903    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.923906    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.924558    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.930987    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.931010    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.931028    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.931034    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931088    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931093    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931096    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931099    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.932922 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931102    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931105    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931108    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931110    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931113    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931116    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931119    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931121    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931124    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931126    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931129    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931132    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931135    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931137    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931140    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931142    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931145    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931147    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931150    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931153    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.933291 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931156    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931158    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931161    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931165    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931169    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931172    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931175    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931178    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931181    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931183    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931186    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931188    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931191    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931194    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931196    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931199    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931202    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931204    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931207    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931209    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.933789 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931212    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931215    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931217    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931220    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931222    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931225    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931228    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931230    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931233    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931235    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931238    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931240    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931243    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931246    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931248    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931252    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931256    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931260    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931264    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.934255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931267    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931271    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931274    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931277    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931279    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931282    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931285    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931288    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931290    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931293    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931295    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931298    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931301    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931303    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931306    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931308    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931311    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931313    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931316    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931318    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.934716 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931321    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931324    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931326    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931329    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931331    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931334    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931336    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931339    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931342    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931345    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931348    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931350    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931353    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931355    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931358    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.935206 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931360    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.931365    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931469    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931473    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931476    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931479    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931481    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931484    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931486    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931488    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931491    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931494    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931496    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931498    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931501    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:37.935559 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931503    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931506    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931508    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931512    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931516    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931519    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931521    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931524    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931526    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931529    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931532    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931535    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931538    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931540    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931543    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931545    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931548    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931550    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931553    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:37.935944 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931555    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931558    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931560    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931562    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931565    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931567    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931570    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931572    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931574    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931577    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931579    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931582    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931584    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931587    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931589    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931592    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931595    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931599    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931601    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:37.936396 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931604    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931606    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931624    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931627    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931630    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931633    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931636    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931638    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931640    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931643    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931645    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931648    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931650    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931653    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931655    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931658    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931660    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931663    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931665    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931667    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:37.936978 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931670    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931672    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931675    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931678    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931680    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931683    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931685    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931688    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931690    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931693    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931695    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931698    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931701    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931703    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931707    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931710    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931713    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931716    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931720    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931722    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:37.937437 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931725    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931727    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931730    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931732    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931735    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931737    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931739    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:37.931742    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.931747    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.931842    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.934406    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.935635    2561 server.go:1019] "Starting client certificate rotation"
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.935727    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:37.937929 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.935758    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:37.938401 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.938388    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:37.939509 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.939482    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:37.947121 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.947107    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:04:37.951263 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.951246    2561 log.go:25] "Validated CRI v1 image API"
Jul 09 18:04:37.953115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.953040    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:04:37.957184 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.957164    2561 fs.go:135] Filesystem UUIDs: map[127e90a1-3606-4448-9e89-22933f1f4481:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 85324f0a-8edd-4c04-878e-046b67780305:/dev/nvme0n1p3]
Jul 09 18:04:37.957247 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.957183    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:04:37.961989 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.961885    2561 manager.go:216] Machine: {Timestamp:2026-07-09 18:04:37.960949291 +0000 UTC m=+0.473758791 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3148812 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2094f62b92ea59295edb9ae18414d7 SystemUUID:ec2094f6-2b92-ea59-295e-db9ae18414d7 BootID:2551a8a5-ad0c-4f8e-b535-43c251f47a85 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4e:85:12:f2:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4e:85:12:f2:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:67:d8:85:27:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:04:37.961989 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.961985    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:04:37.962097 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.962075    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:04:37.962942 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.962918    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:04:37.963073 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.962944    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:04:37.963582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.963572    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:04:37.963627 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.963585    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:04:37.963627 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.963592    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:37.964221 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.964212    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:37.965523 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.965513    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:04:37.965881 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.965864    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:37.969176 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.969165    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:37.969287 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.969278    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:04:37.969321 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.969304    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:04:37.969321 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.969313    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:04:37.969371 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.969326    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:04:37.969371 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.969334    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:04:37.969371 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.969343    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:04:37.969659 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.969648    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:37.969721 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.969667    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:37.971445 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.971423    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:04:37.973302 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973286    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:04:37.973346 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973319    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:04:37.973515 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973493    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:04:37.973515 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973515    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:04:37.973600 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973522    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:04:37.973600 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973527    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:04:37.973600 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973535    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:04:37.973600 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973541    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:04:37.973918 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973909    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:04:37.973918 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973918    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:04:37.973973 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973924    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:04:37.973973 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973930    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:04:37.973973 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973938    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:04:37.973973 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.973946    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:04:37.974681 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.974671    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:04:37.974681 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.974680    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:04:37.977695 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.977681    2561 server.go:1268] "Started kubelet"
Jul 09 18:04:37.977766 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.977735    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:04:37.977876 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.977823    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:04:37.977953 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.977910    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:04:37.978791 ip-10-0-140-229 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:04:37.979143 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.979125    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:04:37.980250 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.980238    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:04:37.983886 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:37.983864    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:04:37.983967 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:37.983925    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:04:37.983967 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:37.983933    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:04:37.984299 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.984285    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-229.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:04:37.984357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.984343    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:04:37.984395 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.984356    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:37.987226 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:37.984169    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-229.ec2.internal.18c0b1315b0ab059  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-229.ec2.internal,UID:ip-10-0-140-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-229.ec2.internal,},FirstTimestamp:2026-07-09 18:04:37.977657433 +0000 UTC m=+0.490466952,LastTimestamp:2026-07-09 18:04:37.977657433 +0000 UTC m=+0.490466952,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-229.ec2.internal,}"
Jul 09 18:04:37.987417 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:37.987398    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:37.987849 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.987815    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:04:37.988324 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.988299    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:04:37.988394 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.988342    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:04:37.988590 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.988578    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:04:37.988691 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.988681    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:04:37.988767 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.988711    2561 factory.go:55] Registering systemd factory
Jul 09 18:04:37.988767 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.988768    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:04:37.989571 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.989553    2561 factory.go:153] Registering CRI-O factory
Jul 09 18:04:37.989571 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.989573    2561 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:04:37.989736 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.989704    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:04:37.990032 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.990014    2561 factory.go:103] Registering Raw factory
Jul 09 18:04:37.990071 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.990045    2561 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:04:37.990419 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.990406    2561 manager.go:349] Starting recovery of all containers
Jul 09 18:04:37.993243 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.993218    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k8ggj"
Jul 09 18:04:37.995770 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:37.995742    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:04:37.996004 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:37.995975    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:04:37.996081 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:37.995995    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:04:38.000390 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.000370    2561 manager.go:354] Recovery completed
Jul 09 18:04:38.002498 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.002475    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k8ggj"
Jul 09 18:04:38.004450 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.004436    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:38.006192 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.006179    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:38.006259 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.006204    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:38.006259 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.006217    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:38.006683 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.006670    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:04:38.006683 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.006678    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:38.006807 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.006681    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:04:38.006807 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.006734    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:38.007543 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.007527    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:38.007626 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.007551    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:38.007626 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.007561    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:38.008372 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:38.008312    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-229.ec2.internal.18c0b1315cbe126a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-229.ec2.internal,UID:ip-10-0-140-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-229.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-229.ec2.internal,},FirstTimestamp:2026-07-09 18:04:38.006190698 +0000 UTC m=+0.519000199,LastTimestamp:2026-07-09 18:04:38.006190698 +0000 UTC m=+0.519000199,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-229.ec2.internal,}"
Jul 09 18:04:38.008784 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.008772    2561 policy_none.go:49] "None policy: Start"
Jul 09 18:04:38.008829 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.008788    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:04:38.008829 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:38.008798    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:04:39.197148 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.197136    2561 policy_none.go:47] "Start"
Jul 09 18:04:39.232643 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.232496    2561 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:04:39.232740 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.232654    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:04:39.232740 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.232668    2561 server.go:87] "Starting device plugin registration server"
Jul 09 18:04:39.235003 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.233131    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:04:39.235003 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.233150    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:04:39.235003 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.233328    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:04:39.235003 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.233410    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:04:39.235003 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.233416    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:04:39.235003 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.234142    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:04:39.235003 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.234199    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:39.307973 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.307931    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:04:39.310125 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.307989    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:04:39.310125 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.308013    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:04:39.310125 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.308054    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:04:39.310668 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.310644    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:39.334881 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.334865    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:39.335683 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.335668    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:39.335752 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.335697    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:39.335752 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.335713    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:39.335752 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.335741    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.344310 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.344294    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.344391 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.344320    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-229.ec2.internal\": node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:39.359545 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.359492    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:39.408624 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.408605    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"]
Jul 09 18:04:39.408683 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.408659    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:39.409973 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.409936    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:39.410029 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.409985    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:39.410029 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.409996    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:39.412273 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.412261    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:39.412377 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.412363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.412434 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.412391    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:39.412874 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.412861    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:39.412874 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.412873    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:39.412999 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.412886    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:39.412999 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.412891    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:39.412999 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.412899    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:39.412999 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.412903    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:39.415267 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.415255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.415312 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.415280    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:39.415905 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.415889    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:39.416010 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.415923    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:39.416010 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.415937    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:39.437497 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.437478    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-229.ec2.internal\" not found" node="ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.441589 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.441574    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-229.ec2.internal\" not found" node="ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.459755 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.459735    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:39.476540 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.476516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/244d0f9322256235dc0c3747942f9000-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"244d0f9322256235dc0c3747942f9000\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.476613 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.476544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/641e9bded5c7fc9e1b94316361d7f097-config\") pod \"kube-apiserver-proxy-ip-10-0-140-229.ec2.internal\" (UID: \"641e9bded5c7fc9e1b94316361d7f097\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.476613 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.476568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/244d0f9322256235dc0c3747942f9000-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"244d0f9322256235dc0c3747942f9000\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.560099 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.560078    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:39.577475 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.577452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/641e9bded5c7fc9e1b94316361d7f097-config\") pod \"kube-apiserver-proxy-ip-10-0-140-229.ec2.internal\" (UID: \"641e9bded5c7fc9e1b94316361d7f097\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.577572 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.577477    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/641e9bded5c7fc9e1b94316361d7f097-config\") pod \"kube-apiserver-proxy-ip-10-0-140-229.ec2.internal\" (UID: \"641e9bded5c7fc9e1b94316361d7f097\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.577572 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.577499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/244d0f9322256235dc0c3747942f9000-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"244d0f9322256235dc0c3747942f9000\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.577572 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.577525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/244d0f9322256235dc0c3747942f9000-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"244d0f9322256235dc0c3747942f9000\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.577572 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.577530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/244d0f9322256235dc0c3747942f9000-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"244d0f9322256235dc0c3747942f9000\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.577704 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.577572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/244d0f9322256235dc0c3747942f9000-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"244d0f9322256235dc0c3747942f9000\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.660876 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.660828    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:39.739369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.739350    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.744349 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:39.744195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:39.760916 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.760890    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:39.861425 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.861399    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:39.961894 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:39.961838    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:40.062413 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:40.062384    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:40.121869 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.121836    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:04:40.122291 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.122016    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:40.163319 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:40.163298    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:40.192446 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.192412    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:59:39 +0000 UTC" deadline="2028-03-09 12:32:05.003613168 +0000 UTC"
Jul 09 18:04:40.192446 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.192441    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14610h27m24.811176511s"
Jul 09 18:04:40.263858 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:40.263799    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:40.364305 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:40.364284    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:40.464783 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:40.464760    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:40.480536 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.480517    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:40.560363 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.560303    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:40.565643 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:40.565626    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Jul 09 18:04:40.583214 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:40.583183    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244d0f9322256235dc0c3747942f9000.slice/crio-f4147f17580f8cd8b0761c32fdd13846173a1285c8bd3ea57f2e55728e354e99 WatchSource:0}: Error finding container f4147f17580f8cd8b0761c32fdd13846173a1285c8bd3ea57f2e55728e354e99: Status 404 returned error can't find the container with id f4147f17580f8cd8b0761c32fdd13846173a1285c8bd3ea57f2e55728e354e99
Jul 09 18:04:40.586993 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.586976    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:04:40.606776 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.606759    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:40.674418 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.674399    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:40.680453 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:40.680434    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641e9bded5c7fc9e1b94316361d7f097.slice/crio-d8ccde132179527d7e763ce1a5b8023fbcdbb94d90d0dc33fba3ba23c31f260e WatchSource:0}: Error finding container d8ccde132179527d7e763ce1a5b8023fbcdbb94d90d0dc33fba3ba23c31f260e: Status 404 returned error can't find the container with id d8ccde132179527d7e763ce1a5b8023fbcdbb94d90d0dc33fba3ba23c31f260e
Jul 09 18:04:40.688345 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.688332    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:40.689149 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.689134    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Jul 09 18:04:40.705327 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.705307    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:40.943820 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:40.943754    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:41.157000 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.156977    2561 apiserver.go:52] "Watching apiserver"
Jul 09 18:04:41.167462 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.167442    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:04:41.168386 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.168365    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-4zdxd","openshift-network-diagnostics/network-check-target-fp6xc","openshift-ovn-kubernetes/ovnkube-node-d8jhz","kube-system/global-pull-secret-syncer-fxrfv","openshift-cluster-node-tuning-operator/tuned-swk86","openshift-dns/node-resolver-gxxk7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal","openshift-multus/multus-7fb5b","openshift-multus/network-metrics-daemon-l5m2b","openshift-network-operator/iptables-alerter-vcgq4","kube-system/konnectivity-agent-25l7d","kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm","openshift-image-registry/node-ca-5jhnd"]
Jul 09 18:04:41.171026 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.171013    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.173090 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.173073    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.174131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.173410    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m6nhv\""
Jul 09 18:04:41.174131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.173917    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:04:41.174131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.173960    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.174131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.174023    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.176050 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.174536    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:41.176523 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.176504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.178706 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.178688    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.178800 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.178754    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:41.180829 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.180811    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.182485 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.182470    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:04:41.183000 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.182986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.183974 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.183955    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:04:41.184054 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.184021    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.184204 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.184191    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.184342 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.184320    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:04:41.184443 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.184330    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tplfg\""
Jul 09 18:04:41.185207 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-run-ovn\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185207 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-cni-bin\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185359 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08138fce-4009-438a-bf62-a17c4f40b74e-env-overrides\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185359 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d62dd174-274a-4a5a-99a3-069c5638ae31-iptables-alerter-script\") pod \"iptables-alerter-vcgq4\" (UID: \"d62dd174-274a-4a5a-99a3-069c5638ae31\") " pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.185359 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185250    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:41.185359 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.185320    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:04:41.185359 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-socket-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185374    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-kubelet\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-slash\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-run-systemd\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185462    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-etc-openvswitch\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f2fr\" (UniqueName: \"kubernetes.io/projected/98c0a723-9d5d-4399-a5ff-3a75d831e611-kube-api-access-5f2fr\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185378    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-log-socket\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-node-log\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrrs\" (UniqueName: \"kubernetes.io/projected/08138fce-4009-438a-bf62-a17c4f40b74e-kube-api-access-nwrrs\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d62dd174-274a-4a5a-99a3-069c5638ae31-host-slash\") pod \"iptables-alerter-vcgq4\" (UID: \"d62dd174-274a-4a5a-99a3-069c5638ae31\") " pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.185582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185580    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185593    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-etc-selinux\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-registration-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185462    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-run-netns\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185700    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-cni-netd\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185745    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rcf8h\""
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185754    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6m5bb\""
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08138fce-4009-438a-bf62-a17c4f40b74e-ovnkube-config\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08138fce-4009-438a-bf62-a17c4f40b74e-ovn-node-metrics-cert\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-device-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-systemd-units\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185875    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185891    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-run-openvswitch\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.185990    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.186006    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-var-lib-openvswitch\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.186011    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-627cp\""
Jul 09 18:04:41.186643 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.186046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186643 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.186072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08138fce-4009-438a-bf62-a17c4f40b74e-ovnkube-script-lib\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.186643 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.186095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9p5p\" (UniqueName: \"kubernetes.io/projected/d62dd174-274a-4a5a-99a3-069c5638ae31-kube-api-access-x9p5p\") pod \"iptables-alerter-vcgq4\" (UID: \"d62dd174-274a-4a5a-99a3-069c5638ae31\") " pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.186643 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.186118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-sys-fs\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.187480 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.187466    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.189843 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.189824    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:41.189968 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.189882    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:41.191432 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.191409    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.191535 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.191462    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.191535 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.191464    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9hpzz\""
Jul 09 18:04:41.191781 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.191763    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:04:41.192231 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.192212    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.194596 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.194552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:04:41.197898 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.197883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.198224 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.198189    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.198321 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.198281    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cgrlv\""
Jul 09 18:04:41.198694 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.198547    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:04:41.198694 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.198558    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.199023 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.198996    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:04:41.199234 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.199210    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:04:41.199415 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.199397    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:04:41.199495 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.199400    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-27n59\""
Jul 09 18:04:41.200852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.200709    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:04:41.200852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.200717    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:04:41.201469 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.201451    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8wvd8\""
Jul 09 18:04:41.201734 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.201717    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:41.238787 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.238761    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xhgnl"
Jul 09 18:04:41.247793 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.247769    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xhgnl"
Jul 09 18:04:41.275134 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.275116    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:04:41.286601 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rzv\" (UniqueName: \"kubernetes.io/projected/bcbf8808-7608-4153-90f4-2dcef7231073-kube-api-access-b4rzv\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.286722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-systemd\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.286722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286627    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:41.286722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ec2f0e2-e620-4920-8313-1a3107dca62a-serviceca\") pod \"node-ca-5jhnd\" (UID: \"9ec2f0e2-e620-4920-8313-1a3107dca62a\") " pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.286722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zmb\" (UniqueName: \"kubernetes.io/projected/9ec2f0e2-e620-4920-8313-1a3107dca62a-kube-api-access-z5zmb\") pod \"node-ca-5jhnd\" (UID: \"9ec2f0e2-e620-4920-8313-1a3107dca62a\") " pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.286722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-run-systemd\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-etc-openvswitch\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-cnibin\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-run-systemd\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-sysctl-conf\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286845    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-etc-openvswitch\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-lib-modules\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-host\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/249ac48c-521a-4576-a202-28091f9257cd-tmp\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gx4h\" (UniqueName: \"kubernetes.io/projected/e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8-kube-api-access-5gx4h\") pod \"node-resolver-gxxk7\" (UID: \"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8\") " pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.286988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-log-socket\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.286995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6cab92a5-eb6e-4d21-b073-f4909c29eee7-agent-certs\") pod \"konnectivity-agent-25l7d\" (UID: \"6cab92a5-eb6e-4d21-b073-f4909c29eee7\") " pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-log-socket\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287030    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-var-lib-cni-multus\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-multus-conf-dir\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcbf8808-7608-4153-90f4-2dcef7231073-multus-daemon-config\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-etc-kubernetes\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-sysconfig\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-node-log\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-etc-selinux\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-cnibin\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-node-log\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-etc-selinux\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.287405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-registration-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287333    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-sys\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghlxf\" (UniqueName: \"kubernetes.io/projected/641493c8-16d5-46c0-bd0a-8763e005dcdf-kube-api-access-ghlxf\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-run-netns\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08138fce-4009-438a-bf62-a17c4f40b74e-ovnkube-config\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-registration-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-run-netns\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3555656-954a-49c5-a49c-50c44b2f7ed0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3555656-954a-49c5-a49c-50c44b2f7ed0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-systemd-units\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-run-openvswitch\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3555656-954a-49c5-a49c-50c44b2f7ed0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcbf8808-7608-4153-90f4-2dcef7231073-cni-binary-copy\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-systemd-units\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-run-openvswitch\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.288217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-run-netns\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-modprobe-d\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/249ac48c-521a-4576-a202-28091f9257cd-etc-tuned\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-sys-fs\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-system-cni-dir\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-multus-cni-dir\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-var-lib-kubelet\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-sys-fs\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-var-lib-kubelet\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ec2f0e2-e620-4920-8313-1a3107dca62a-host\") pod \"node-ca-5jhnd\" (UID: \"9ec2f0e2-e620-4920-8313-1a3107dca62a\") " pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-run-ovn\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-cni-bin\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08138fce-4009-438a-bf62-a17c4f40b74e-env-overrides\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-socket-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-run-ovn\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-multus-socket-dir-parent\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-cni-bin\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.287967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08138fce-4009-438a-bf62-a17c4f40b74e-ovnkube-config\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288017    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d52ad56-9937-4f10-88c3-051fe2f35027-kubelet-config\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-kubelet\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-slash\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-kubelet\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-socket-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f2fr\" (UniqueName: \"kubernetes.io/projected/98c0a723-9d5d-4399-a5ff-3a75d831e611-kube-api-access-5f2fr\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-run-k8s-cni-cncf-io\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-run\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288160    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx676\" (UniqueName: \"kubernetes.io/projected/249ac48c-521a-4576-a202-28091f9257cd-kube-api-access-vx676\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8-hosts-file\") pod \"node-resolver-gxxk7\" (UID: \"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8\") " pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-sysctl-d\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-slash\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrrs\" (UniqueName: \"kubernetes.io/projected/08138fce-4009-438a-bf62-a17c4f40b74e-kube-api-access-nwrrs\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d62dd174-274a-4a5a-99a3-069c5638ae31-host-slash\") pod \"iptables-alerter-vcgq4\" (UID: \"d62dd174-274a-4a5a-99a3-069c5638ae31\") " pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.289722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6cab92a5-eb6e-4d21-b073-f4909c29eee7-konnectivity-ca\") pod \"konnectivity-agent-25l7d\" (UID: \"6cab92a5-eb6e-4d21-b073-f4909c29eee7\") " pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-os-release\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d62dd174-274a-4a5a-99a3-069c5638ae31-host-slash\") pod \"iptables-alerter-vcgq4\" (UID: \"d62dd174-274a-4a5a-99a3-069c5638ae31\") " pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-system-cni-dir\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d52ad56-9937-4f10-88c3-051fe2f35027-dbus\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8-tmp-dir\") pod \"node-resolver-gxxk7\" (UID: \"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8\") " pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-cni-netd\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08138fce-4009-438a-bf62-a17c4f40b74e-env-overrides\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08138fce-4009-438a-bf62-a17c4f40b74e-ovn-node-metrics-cert\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-device-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-host-cni-netd\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-hostroot\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288538    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-var-lib-cni-bin\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-var-lib-openvswitch\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98c0a723-9d5d-4399-a5ff-3a75d831e611-device-dir\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08138fce-4009-438a-bf62-a17c4f40b74e-ovnkube-script-lib\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9p5p\" (UniqueName: \"kubernetes.io/projected/d62dd174-274a-4a5a-99a3-069c5638ae31-kube-api-access-x9p5p\") pod \"iptables-alerter-vcgq4\" (UID: \"d62dd174-274a-4a5a-99a3-069c5638ae31\") " pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.290369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:41.291093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288628    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08138fce-4009-438a-bf62-a17c4f40b74e-var-lib-openvswitch\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.291093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288658    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-os-release\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.291093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-kubernetes\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.291093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d62dd174-274a-4a5a-99a3-069c5638ae31-iptables-alerter-script\") pod \"iptables-alerter-vcgq4\" (UID: \"d62dd174-274a-4a5a-99a3-069c5638ae31\") " pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.291093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288749    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:04:41.291093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288809    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kt9f\" (UniqueName: \"kubernetes.io/projected/a3555656-954a-49c5-a49c-50c44b2f7ed0-kube-api-access-6kt9f\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.291093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.288861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-run-multus-certs\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.291093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.289131    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08138fce-4009-438a-bf62-a17c4f40b74e-ovnkube-script-lib\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.291093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.289258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d62dd174-274a-4a5a-99a3-069c5638ae31-iptables-alerter-script\") pod \"iptables-alerter-vcgq4\" (UID: \"d62dd174-274a-4a5a-99a3-069c5638ae31\") " pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.292236 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.292218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08138fce-4009-438a-bf62-a17c4f40b74e-ovn-node-metrics-cert\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.298096 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.298078    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f2fr\" (UniqueName: \"kubernetes.io/projected/98c0a723-9d5d-4399-a5ff-3a75d831e611-kube-api-access-5f2fr\") pod \"aws-ebs-csi-driver-node-w5qpm\" (UID: \"98c0a723-9d5d-4399-a5ff-3a75d831e611\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.298198 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.298122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrrs\" (UniqueName: \"kubernetes.io/projected/08138fce-4009-438a-bf62-a17c4f40b74e-kube-api-access-nwrrs\") pod \"ovnkube-node-d8jhz\" (UID: \"08138fce-4009-438a-bf62-a17c4f40b74e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.298986 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.298969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9p5p\" (UniqueName: \"kubernetes.io/projected/d62dd174-274a-4a5a-99a3-069c5638ae31-kube-api-access-x9p5p\") pod \"iptables-alerter-vcgq4\" (UID: \"d62dd174-274a-4a5a-99a3-069c5638ae31\") " pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.312794 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.312753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal" event={"ID":"641e9bded5c7fc9e1b94316361d7f097","Type":"ContainerStarted","Data":"d8ccde132179527d7e763ce1a5b8023fbcdbb94d90d0dc33fba3ba23c31f260e"}
Jul 09 18:04:41.314084 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.314065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal" event={"ID":"244d0f9322256235dc0c3747942f9000","Type":"ContainerStarted","Data":"f4147f17580f8cd8b0761c32fdd13846173a1285c8bd3ea57f2e55728e354e99"}
Jul 09 18:04:41.390144 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-cnibin\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390144 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-sysctl-conf\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.390357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-lib-modules\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.390357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-host\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.390357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/249ac48c-521a-4576-a202-28091f9257cd-tmp\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.390357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gx4h\" (UniqueName: \"kubernetes.io/projected/e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8-kube-api-access-5gx4h\") pod \"node-resolver-gxxk7\" (UID: \"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8\") " pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.390357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6cab92a5-eb6e-4d21-b073-f4909c29eee7-agent-certs\") pod \"konnectivity-agent-25l7d\" (UID: \"6cab92a5-eb6e-4d21-b073-f4909c29eee7\") " pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:04:41.390357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-var-lib-cni-multus\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-multus-conf-dir\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcbf8808-7608-4153-90f4-2dcef7231073-multus-daemon-config\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-etc-kubernetes\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-sysconfig\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-cnibin\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-sys\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghlxf\" (UniqueName: \"kubernetes.io/projected/641493c8-16d5-46c0-bd0a-8763e005dcdf-kube-api-access-ghlxf\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3555656-954a-49c5-a49c-50c44b2f7ed0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3555656-954a-49c5-a49c-50c44b2f7ed0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3555656-954a-49c5-a49c-50c44b2f7ed0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcbf8808-7608-4153-90f4-2dcef7231073-cni-binary-copy\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-run-netns\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-modprobe-d\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/249ac48c-521a-4576-a202-28091f9257cd-etc-tuned\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-system-cni-dir\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-cnibin\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-multus-cni-dir\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.390826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-var-lib-kubelet\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-var-lib-kubelet\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ec2f0e2-e620-4920-8313-1a3107dca62a-host\") pod \"node-ca-5jhnd\" (UID: \"9ec2f0e2-e620-4920-8313-1a3107dca62a\") " pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-sys\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390900    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ec2f0e2-e620-4920-8313-1a3107dca62a-host\") pod \"node-ca-5jhnd\" (UID: \"9ec2f0e2-e620-4920-8313-1a3107dca62a\") " pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390904    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-multus-socket-dir-parent\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-multus-cni-dir\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-cnibin\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-etc-kubernetes\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-run-netns\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391015    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-var-lib-kubelet\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d52ad56-9937-4f10-88c3-051fe2f35027-kubelet-config\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-var-lib-kubelet\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391075    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-multus-conf-dir\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-sysconfig\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-modprobe-d\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.391146    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.391192    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret podName:5d52ad56-9937-4f10-88c3-051fe2f35027 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:41.891177832 +0000 UTC m=+3.217653517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret") pod "global-pull-secret-syncer-fxrfv" (UID: "5d52ad56-9937-4f10-88c3-051fe2f35027") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:41.391605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391630    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcbf8808-7608-4153-90f4-2dcef7231073-multus-daemon-config\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3555656-954a-49c5-a49c-50c44b2f7ed0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-system-cni-dir\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-var-lib-cni-multus\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.390937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d52ad56-9937-4f10-88c3-051fe2f35027-kubelet-config\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-run-k8s-cni-cncf-io\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-run\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx676\" (UniqueName: \"kubernetes.io/projected/249ac48c-521a-4576-a202-28091f9257cd-kube-api-access-vx676\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8-hosts-file\") pod \"node-resolver-gxxk7\" (UID: \"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8\") " pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-sysctl-d\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6cab92a5-eb6e-4d21-b073-f4909c29eee7-konnectivity-ca\") pod \"konnectivity-agent-25l7d\" (UID: \"6cab92a5-eb6e-4d21-b073-f4909c29eee7\") " pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-os-release\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-system-cni-dir\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391967    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d52ad56-9937-4f10-88c3-051fe2f35027-dbus\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.391994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8-tmp-dir\") pod \"node-resolver-gxxk7\" (UID: \"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8\") " pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-hostroot\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-var-lib-cni-bin\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.392464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-os-release\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-kubernetes\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kt9f\" (UniqueName: \"kubernetes.io/projected/a3555656-954a-49c5-a49c-50c44b2f7ed0-kube-api-access-6kt9f\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcbf8808-7608-4153-90f4-2dcef7231073-cni-binary-copy\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3555656-954a-49c5-a49c-50c44b2f7ed0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392224    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-run\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-run-k8s-cni-cncf-io\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-run-multus-certs\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-run-multus-certs\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rzv\" (UniqueName: \"kubernetes.io/projected/bcbf8808-7608-4153-90f4-2dcef7231073-kube-api-access-b4rzv\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8-hosts-file\") pod \"node-resolver-gxxk7\" (UID: \"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8\") " pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-systemd\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392356    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ec2f0e2-e620-4920-8313-1a3107dca62a-serviceca\") pod \"node-ca-5jhnd\" (UID: \"9ec2f0e2-e620-4920-8313-1a3107dca62a\") " pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zmb\" (UniqueName: \"kubernetes.io/projected/9ec2f0e2-e620-4920-8313-1a3107dca62a-kube-api-access-z5zmb\") pod \"node-ca-5jhnd\" (UID: \"9ec2f0e2-e620-4920-8313-1a3107dca62a\") " pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392389    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-sysctl-d\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.393298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-system-cni-dir\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392462    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d52ad56-9937-4f10-88c3-051fe2f35027-dbus\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-multus-socket-dir-parent\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392709    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-hostroot\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-host-var-lib-cni-bin\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392745    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8-tmp-dir\") pod \"node-resolver-gxxk7\" (UID: \"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8\") " pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3555656-954a-49c5-a49c-50c44b2f7ed0-os-release\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-lib-modules\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3555656-954a-49c5-a49c-50c44b2f7ed0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-kubernetes\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.392926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-sysctl-conf\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.393003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bcbf8808-7608-4153-90f4-2dcef7231073-os-release\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.393188    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/249ac48c-521a-4576-a202-28091f9257cd-tmp\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.393211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-etc-systemd\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.393252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/249ac48c-521a-4576-a202-28091f9257cd-host\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.393573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6cab92a5-eb6e-4d21-b073-f4909c29eee7-konnectivity-ca\") pod \"konnectivity-agent-25l7d\" (UID: \"6cab92a5-eb6e-4d21-b073-f4909c29eee7\") " pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:04:41.394095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.393676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ec2f0e2-e620-4920-8313-1a3107dca62a-serviceca\") pod \"node-ca-5jhnd\" (UID: \"9ec2f0e2-e620-4920-8313-1a3107dca62a\") " pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.394833 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.394122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6cab92a5-eb6e-4d21-b073-f4909c29eee7-agent-certs\") pod \"konnectivity-agent-25l7d\" (UID: \"6cab92a5-eb6e-4d21-b073-f4909c29eee7\") " pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:04:41.396234 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.396212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/249ac48c-521a-4576-a202-28091f9257cd-etc-tuned\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.396444 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.396425    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:41.396512 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.396494    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs podName:641493c8-16d5-46c0-bd0a-8763e005dcdf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:41.896476668 +0000 UTC m=+3.222952369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs") pod "network-metrics-daemon-l5m2b" (UID: "641493c8-16d5-46c0-bd0a-8763e005dcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:41.413424 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.413404    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:41.413424 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.413425    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:41.413424 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.413437    2561 projected.go:196] Error preparing data for projected volume kube-api-access-wgxgn for pod openshift-network-diagnostics/network-check-target-fp6xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:41.413687 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.413483    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn podName:1f662f2f-4128-4b2d-be2a-e3053a6f47e5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:41.913471302 +0000 UTC m=+3.239946992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wgxgn" (UniqueName: "kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn") pod "network-check-target-fp6xc" (UID: "1f662f2f-4128-4b2d-be2a-e3053a6f47e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:41.417099 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.417003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rzv\" (UniqueName: \"kubernetes.io/projected/bcbf8808-7608-4153-90f4-2dcef7231073-kube-api-access-b4rzv\") pod \"multus-7fb5b\" (UID: \"bcbf8808-7608-4153-90f4-2dcef7231073\") " pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.419856 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.419781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gx4h\" (UniqueName: \"kubernetes.io/projected/e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8-kube-api-access-5gx4h\") pod \"node-resolver-gxxk7\" (UID: \"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8\") " pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.424295 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.423731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kt9f\" (UniqueName: \"kubernetes.io/projected/a3555656-954a-49c5-a49c-50c44b2f7ed0-kube-api-access-6kt9f\") pod \"multus-additional-cni-plugins-4zdxd\" (UID: \"a3555656-954a-49c5-a49c-50c44b2f7ed0\") " pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.424295 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.424244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghlxf\" (UniqueName: \"kubernetes.io/projected/641493c8-16d5-46c0-bd0a-8763e005dcdf-kube-api-access-ghlxf\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:41.425167 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.425144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx676\" (UniqueName: \"kubernetes.io/projected/249ac48c-521a-4576-a202-28091f9257cd-kube-api-access-vx676\") pod \"tuned-swk86\" (UID: \"249ac48c-521a-4576-a202-28091f9257cd\") " pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.425681 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.425643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zmb\" (UniqueName: \"kubernetes.io/projected/9ec2f0e2-e620-4920-8313-1a3107dca62a-kube-api-access-z5zmb\") pod \"node-ca-5jhnd\" (UID: \"9ec2f0e2-e620-4920-8313-1a3107dca62a\") " pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.482678 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.482614    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vcgq4"
Jul 09 18:04:41.490250 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.490227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm"
Jul 09 18:04:41.498424 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:41.498399    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62dd174_274a_4a5a_99a3_069c5638ae31.slice/crio-1f684d419584348df873e8638c4a86ea3ec0ed6388660f7c735948ff6d45f83d WatchSource:0}: Error finding container 1f684d419584348df873e8638c4a86ea3ec0ed6388660f7c735948ff6d45f83d: Status 404 returned error can't find the container with id 1f684d419584348df873e8638c4a86ea3ec0ed6388660f7c735948ff6d45f83d
Jul 09 18:04:41.499240 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.499067    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxxk7"
Jul 09 18:04:41.504611 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.504449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:04:41.511227 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:41.510931    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c0a723_9d5d_4399_a5ff_3a75d831e611.slice/crio-5f69ba74c4751df7c86314cb9393d3b3fa3b2b63a237a3669d7821f5534405fd WatchSource:0}: Error finding container 5f69ba74c4751df7c86314cb9393d3b3fa3b2b63a237a3669d7821f5534405fd: Status 404 returned error can't find the container with id 5f69ba74c4751df7c86314cb9393d3b3fa3b2b63a237a3669d7821f5534405fd
Jul 09 18:04:41.511227 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.511065    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-swk86"
Jul 09 18:04:41.511615 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:41.511595    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b578cd_ec3f_4c6c_8acb_7fe9818c4bf8.slice/crio-55aed6bae2bd1a88e9245c1023c4f73e28d78d5b24842427f68c4468d5b920bd WatchSource:0}: Error finding container 55aed6bae2bd1a88e9245c1023c4f73e28d78d5b24842427f68c4468d5b920bd: Status 404 returned error can't find the container with id 55aed6bae2bd1a88e9245c1023c4f73e28d78d5b24842427f68c4468d5b920bd
Jul 09 18:04:41.517566 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.517546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5jhnd"
Jul 09 18:04:41.520429 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:41.520316    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08138fce_4009_438a_bf62_a17c4f40b74e.slice/crio-2eee4fe1ee9d631aad78d1bb61bf2d830371855246a7798eb4d2b58481f54b09 WatchSource:0}: Error finding container 2eee4fe1ee9d631aad78d1bb61bf2d830371855246a7798eb4d2b58481f54b09: Status 404 returned error can't find the container with id 2eee4fe1ee9d631aad78d1bb61bf2d830371855246a7798eb4d2b58481f54b09
Jul 09 18:04:41.523842 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.523825    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4zdxd"
Jul 09 18:04:41.524456 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:41.524430    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249ac48c_521a_4576_a202_28091f9257cd.slice/crio-2f39efb4973c2d419106cca2dc1876f269ccbcbfad39bd8f5896f10c29b5cd01 WatchSource:0}: Error finding container 2f39efb4973c2d419106cca2dc1876f269ccbcbfad39bd8f5896f10c29b5cd01: Status 404 returned error can't find the container with id 2f39efb4973c2d419106cca2dc1876f269ccbcbfad39bd8f5896f10c29b5cd01
Jul 09 18:04:41.529806 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.529787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:04:41.532518 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:41.532495    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec2f0e2_e620_4920_8313_1a3107dca62a.slice/crio-89db55c96937d7eb016cd3faf5f28556e71a31eeee4e27802f3ced3555804f3f WatchSource:0}: Error finding container 89db55c96937d7eb016cd3faf5f28556e71a31eeee4e27802f3ced3555804f3f: Status 404 returned error can't find the container with id 89db55c96937d7eb016cd3faf5f28556e71a31eeee4e27802f3ced3555804f3f
Jul 09 18:04:41.535264 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.535243    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7fb5b"
Jul 09 18:04:41.536255 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:41.536232    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3555656_954a_49c5_a49c_50c44b2f7ed0.slice/crio-a9a201cb77b6060a5b89e47b4c234548194450338de18c927e1b61c4f76d08c2 WatchSource:0}: Error finding container a9a201cb77b6060a5b89e47b4c234548194450338de18c927e1b61c4f76d08c2: Status 404 returned error can't find the container with id a9a201cb77b6060a5b89e47b4c234548194450338de18c927e1b61c4f76d08c2
Jul 09 18:04:41.546046 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:41.546012    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cab92a5_eb6e_4d21_b073_f4909c29eee7.slice/crio-760194657c44a24919514573facee49c0fc8bc01fc258dfee560b9ae25caa243 WatchSource:0}: Error finding container 760194657c44a24919514573facee49c0fc8bc01fc258dfee560b9ae25caa243: Status 404 returned error can't find the container with id 760194657c44a24919514573facee49c0fc8bc01fc258dfee560b9ae25caa243
Jul 09 18:04:41.550669 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:04:41.550643    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbf8808_7608_4153_90f4_2dcef7231073.slice/crio-746466496b238664b94a1e321d45a2c1a662428b2ab14a2dbe19c0fea278c4fd WatchSource:0}: Error finding container 746466496b238664b94a1e321d45a2c1a662428b2ab14a2dbe19c0fea278c4fd: Status 404 returned error can't find the container with id 746466496b238664b94a1e321d45a2c1a662428b2ab14a2dbe19c0fea278c4fd
Jul 09 18:04:41.896802 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.896717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:41.896802 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.896761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:41.897022 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.896863    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:41.897022 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.896875    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:41.897022 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.896912    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret podName:5d52ad56-9937-4f10-88c3-051fe2f35027 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.896899488 +0000 UTC m=+4.223375173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret") pod "global-pull-secret-syncer-fxrfv" (UID: "5d52ad56-9937-4f10-88c3-051fe2f35027") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:41.897022 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.896934    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs podName:641493c8-16d5-46c0-bd0a-8763e005dcdf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.89691799 +0000 UTC m=+4.223393690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs") pod "network-metrics-daemon-l5m2b" (UID: "641493c8-16d5-46c0-bd0a-8763e005dcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:41.997355 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:41.997322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:41.997531 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.997498    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:41.997531 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.997520    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:41.997644 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.997533    2561 projected.go:196] Error preparing data for projected volume kube-api-access-wgxgn for pod openshift-network-diagnostics/network-check-target-fp6xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:41.997644 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:41.997595    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn podName:1f662f2f-4128-4b2d-be2a-e3053a6f47e5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.99757612 +0000 UTC m=+4.324051811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgxgn" (UniqueName: "kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn") pod "network-check-target-fp6xc" (UID: "1f662f2f-4128-4b2d-be2a-e3053a6f47e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:42.251368 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.251254    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:41 +0000 UTC" deadline="2028-01-11 21:05:56.21343794 +0000 UTC"
Jul 09 18:04:42.251368 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.251292    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13227h1m13.962149905s"
Jul 09 18:04:42.318046 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.317912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7fb5b" event={"ID":"bcbf8808-7608-4153-90f4-2dcef7231073","Type":"ContainerStarted","Data":"746466496b238664b94a1e321d45a2c1a662428b2ab14a2dbe19c0fea278c4fd"}
Jul 09 18:04:42.319350 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.319282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zdxd" event={"ID":"a3555656-954a-49c5-a49c-50c44b2f7ed0","Type":"ContainerStarted","Data":"a9a201cb77b6060a5b89e47b4c234548194450338de18c927e1b61c4f76d08c2"}
Jul 09 18:04:42.321666 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.321620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5jhnd" event={"ID":"9ec2f0e2-e620-4920-8313-1a3107dca62a","Type":"ContainerStarted","Data":"89db55c96937d7eb016cd3faf5f28556e71a31eeee4e27802f3ced3555804f3f"}
Jul 09 18:04:42.323813 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.323737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vcgq4" event={"ID":"d62dd174-274a-4a5a-99a3-069c5638ae31","Type":"ContainerStarted","Data":"1f684d419584348df873e8638c4a86ea3ec0ed6388660f7c735948ff6d45f83d"}
Jul 09 18:04:42.324878 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.324824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-25l7d" event={"ID":"6cab92a5-eb6e-4d21-b073-f4909c29eee7","Type":"ContainerStarted","Data":"760194657c44a24919514573facee49c0fc8bc01fc258dfee560b9ae25caa243"}
Jul 09 18:04:42.326315 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.326256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-swk86" event={"ID":"249ac48c-521a-4576-a202-28091f9257cd","Type":"ContainerStarted","Data":"2f39efb4973c2d419106cca2dc1876f269ccbcbfad39bd8f5896f10c29b5cd01"}
Jul 09 18:04:42.327442 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.327418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerStarted","Data":"2eee4fe1ee9d631aad78d1bb61bf2d830371855246a7798eb4d2b58481f54b09"}
Jul 09 18:04:42.328605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.328583    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxxk7" event={"ID":"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8","Type":"ContainerStarted","Data":"55aed6bae2bd1a88e9245c1023c4f73e28d78d5b24842427f68c4468d5b920bd"}
Jul 09 18:04:42.330090 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.330040    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm" event={"ID":"98c0a723-9d5d-4399-a5ff-3a75d831e611","Type":"ContainerStarted","Data":"5f69ba74c4751df7c86314cb9393d3b3fa3b2b63a237a3669d7821f5534405fd"}
Jul 09 18:04:42.903002 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.902910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:42.903002 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:42.902987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:42.903213 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:42.903169    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:42.903270 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:42.903250    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret podName:5d52ad56-9937-4f10-88c3-051fe2f35027 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:44.903217066 +0000 UTC m=+6.229692769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret") pod "global-pull-secret-syncer-fxrfv" (UID: "5d52ad56-9937-4f10-88c3-051fe2f35027") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:42.903339 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:42.903297    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:42.903391 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:42.903361    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs podName:641493c8-16d5-46c0-bd0a-8763e005dcdf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:44.903344062 +0000 UTC m=+6.229819749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs") pod "network-metrics-daemon-l5m2b" (UID: "641493c8-16d5-46c0-bd0a-8763e005dcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:43.004518 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:43.003886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:43.004518 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:43.004110    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:43.004518 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:43.004132    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:43.004518 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:43.004145    2561 projected.go:196] Error preparing data for projected volume kube-api-access-wgxgn for pod openshift-network-diagnostics/network-check-target-fp6xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:43.004518 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:43.004201    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn podName:1f662f2f-4128-4b2d-be2a-e3053a6f47e5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:45.004184664 +0000 UTC m=+6.330660364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgxgn" (UniqueName: "kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn") pod "network-check-target-fp6xc" (UID: "1f662f2f-4128-4b2d-be2a-e3053a6f47e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:43.252260 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:43.252025    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:41 +0000 UTC" deadline="2027-12-07 06:40:00.944147655 +0000 UTC"
Jul 09 18:04:43.252648 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:43.252263    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12372h35m17.691890691s"
Jul 09 18:04:43.310056 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:43.309288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:43.310056 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:43.309312    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:43.310056 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:43.309413    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:43.310056 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:43.309656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:43.310056 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:43.309746    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:43.310056 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:43.309827    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:04:43.339267 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:43.339226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal" event={"ID":"641e9bded5c7fc9e1b94316361d7f097","Type":"ContainerStarted","Data":"6f06880516616dfdab2797ee3c044d13cbbd1d2e72cbc9b8866034a4da2af195"}
Jul 09 18:04:44.357523 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:44.357486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal" event={"ID":"244d0f9322256235dc0c3747942f9000","Type":"ContainerStarted","Data":"d203bb2cd2e699c9a3cb44bb00cb177c1afb0403248edac4ed2c3f56745371b2"}
Jul 09 18:04:44.372927 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:44.372882    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal" podStartSLOduration=4.372867539 podStartE2EDuration="4.372867539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:43.35467033 +0000 UTC m=+4.681146038" watchObservedRunningTime="2026-07-09 18:04:44.372867539 +0000 UTC m=+5.699343292"
Jul 09 18:04:44.927505 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:44.926745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:44.927505 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:44.926805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:44.927505 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:44.926934    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:44.927505 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:44.927007    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret podName:5d52ad56-9937-4f10-88c3-051fe2f35027 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:48.926990588 +0000 UTC m=+10.253466285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret") pod "global-pull-secret-syncer-fxrfv" (UID: "5d52ad56-9937-4f10-88c3-051fe2f35027") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:44.927505 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:44.927412    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:44.927505 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:44.927467    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs podName:641493c8-16d5-46c0-bd0a-8763e005dcdf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:48.927450464 +0000 UTC m=+10.253926168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs") pod "network-metrics-daemon-l5m2b" (UID: "641493c8-16d5-46c0-bd0a-8763e005dcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:45.027844 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:45.027806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:45.028048 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:45.028035    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:45.028115 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:45.028057    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:45.028115 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:45.028071    2561 projected.go:196] Error preparing data for projected volume kube-api-access-wgxgn for pod openshift-network-diagnostics/network-check-target-fp6xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:45.028210 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:45.028133    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn podName:1f662f2f-4128-4b2d-be2a-e3053a6f47e5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:49.028113984 +0000 UTC m=+10.354589672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgxgn" (UniqueName: "kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn") pod "network-check-target-fp6xc" (UID: "1f662f2f-4128-4b2d-be2a-e3053a6f47e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:45.308853 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:45.308772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:45.308853 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:45.308813    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:45.309044 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:45.308772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:45.309044 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:45.308931    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:04:45.309044 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:45.309017    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:45.309161 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:45.309084    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:46.365174 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:46.365142    2561 generic.go:356] "Generic (PLEG): container finished" podID="244d0f9322256235dc0c3747942f9000" containerID="d203bb2cd2e699c9a3cb44bb00cb177c1afb0403248edac4ed2c3f56745371b2" exitCode=0
Jul 09 18:04:46.365621 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:46.365193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal" event={"ID":"244d0f9322256235dc0c3747942f9000","Type":"ContainerDied","Data":"d203bb2cd2e699c9a3cb44bb00cb177c1afb0403248edac4ed2c3f56745371b2"}
Jul 09 18:04:47.309287 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:47.309251    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:47.309287 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:47.309282    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:47.309514 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:47.309252    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:47.309514 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:47.309392    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:04:47.309514 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:47.309475    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:47.309649 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:47.309572    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:48.958866 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:48.958217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:48.958866 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:48.958283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:48.958866 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:48.958422    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:48.958866 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:48.958483    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret podName:5d52ad56-9937-4f10-88c3-051fe2f35027 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:56.958465241 +0000 UTC m=+18.284940947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret") pod "global-pull-secret-syncer-fxrfv" (UID: "5d52ad56-9937-4f10-88c3-051fe2f35027") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:48.958866 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:48.958767    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:48.958866 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:48.958863    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs podName:641493c8-16d5-46c0-bd0a-8763e005dcdf nodeName:}" failed. No retries permitted until 2026-07-09 18:04:56.958828281 +0000 UTC m=+18.285303971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs") pod "network-metrics-daemon-l5m2b" (UID: "641493c8-16d5-46c0-bd0a-8763e005dcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:49.059201 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:49.059163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:49.059355 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:49.059320    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:49.059355 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:49.059338    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:49.059355 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:49.059350    2561 projected.go:196] Error preparing data for projected volume kube-api-access-wgxgn for pod openshift-network-diagnostics/network-check-target-fp6xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:49.059517 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:49.059401    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn podName:1f662f2f-4128-4b2d-be2a-e3053a6f47e5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:57.059383003 +0000 UTC m=+18.385858693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgxgn" (UniqueName: "kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn") pod "network-check-target-fp6xc" (UID: "1f662f2f-4128-4b2d-be2a-e3053a6f47e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:49.309595 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:49.309516    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:49.309740 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:49.309620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:49.309740 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:49.309625    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:49.309832 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:49.309753    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:04:49.309832 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:49.309785    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:49.309900 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:49.309851    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:51.308273 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:51.308239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:51.308273 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:51.308256    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:51.308760 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:51.308239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:51.308760 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:51.308376    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:04:51.308760 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:51.308466    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:51.308760 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:51.308542    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:53.308959 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:53.308908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:53.308959 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:53.308919    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:53.309374 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:53.309053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:53.309374 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:53.309070    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:04:53.309374 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:53.309168    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:53.309374 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:53.309237    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:55.308814 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:55.308775    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:55.308814 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:55.308805    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:55.309341 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:55.308774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:55.309341 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:55.308885    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:04:55.309341 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:55.308980    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:55.309341 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:55.309044    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:57.025507 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:57.025474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:57.025984 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:57.025518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:57.025984 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.025635    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:57.025984 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.025647    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:57.025984 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.025698    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret podName:5d52ad56-9937-4f10-88c3-051fe2f35027 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.025678701 +0000 UTC m=+34.352154390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret") pod "global-pull-secret-syncer-fxrfv" (UID: "5d52ad56-9937-4f10-88c3-051fe2f35027") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:57.025984 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.025713    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs podName:641493c8-16d5-46c0-bd0a-8763e005dcdf nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.025706148 +0000 UTC m=+34.352181834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs") pod "network-metrics-daemon-l5m2b" (UID: "641493c8-16d5-46c0-bd0a-8763e005dcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:57.126310 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:57.126272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:57.126495 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.126463    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:57.126495 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.126484    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:57.126495 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.126496    2561 projected.go:196] Error preparing data for projected volume kube-api-access-wgxgn for pod openshift-network-diagnostics/network-check-target-fp6xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:57.126645 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.126557    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn podName:1f662f2f-4128-4b2d-be2a-e3053a6f47e5 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.126539444 +0000 UTC m=+34.453015133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgxgn" (UniqueName: "kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn") pod "network-check-target-fp6xc" (UID: "1f662f2f-4128-4b2d-be2a-e3053a6f47e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:57.308965 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:57.308877    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:57.309112 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:57.308877    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:57.309112 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.309009    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:57.309112 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:57.308882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:57.309112 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.309076    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:57.309333 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:57.309171    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:04:59.309754 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:59.309721    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:04:59.310202 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:59.309820    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:04:59.310202 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:59.309859    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:04:59.310202 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:04:59.309962    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:04:59.310202 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:59.309958    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:04:59.310202 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:04:59.310065    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:05:01.308931 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.308562    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:01.309805 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.308566    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:01.309805 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:01.309046    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:05:01.309805 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.308613    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:01.309805 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:01.309089    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:05:01.309805 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:01.309213    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:05:01.392198 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.392172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm" event={"ID":"98c0a723-9d5d-4399-a5ff-3a75d831e611","Type":"ContainerStarted","Data":"3560da21b65721d2d990df7e1014bf3f664d8d02dd68b7b08469644b9a06adf1"}
Jul 09 18:05:01.394062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.394041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal" event={"ID":"244d0f9322256235dc0c3747942f9000","Type":"ContainerStarted","Data":"374b4777e33064a15261f299bb9d13c62baf417095f1a18c77fae9181da0d7a6"}
Jul 09 18:05:01.395562 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.395533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7fb5b" event={"ID":"bcbf8808-7608-4153-90f4-2dcef7231073","Type":"ContainerStarted","Data":"840450235cd83de1a4ca5a124fa36fab7e3c688e5e2021c157b35e7e2d0a8614"}
Jul 09 18:05:01.397045 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.397024    2561 generic.go:356] "Generic (PLEG): container finished" podID="a3555656-954a-49c5-a49c-50c44b2f7ed0" containerID="9bc3e3566c0439edc36fcfa4c5eb97fd56929c41cbb4d7b04dad4acec82eb816" exitCode=0
Jul 09 18:05:01.397125 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.397087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zdxd" event={"ID":"a3555656-954a-49c5-a49c-50c44b2f7ed0","Type":"ContainerDied","Data":"9bc3e3566c0439edc36fcfa4c5eb97fd56929c41cbb4d7b04dad4acec82eb816"}
Jul 09 18:05:01.398975 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.398938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5jhnd" event={"ID":"9ec2f0e2-e620-4920-8313-1a3107dca62a","Type":"ContainerStarted","Data":"71ba09b0b1461684996d31d8d38d5c5759504289a091382364572374f0487e19"}
Jul 09 18:05:01.400397 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.400378    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-25l7d" event={"ID":"6cab92a5-eb6e-4d21-b073-f4909c29eee7","Type":"ContainerStarted","Data":"99665ca06ba90c5c74f4a80f889db7ff203f1ee5943b6549de86c9b4737c1bfe"}
Jul 09 18:05:01.402063 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.402037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-swk86" event={"ID":"249ac48c-521a-4576-a202-28091f9257cd","Type":"ContainerStarted","Data":"bd8019b9f13f71a1cbccc66ba5394ab7d9709904f0ee18d36b0e347035586ce1"}
Jul 09 18:05:01.404890 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.404871    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:05:01.405303 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.405278    2561 generic.go:356] "Generic (PLEG): container finished" podID="08138fce-4009-438a-bf62-a17c4f40b74e" containerID="b158288cf19628b62e27723dad326fad4491649d81c1acd35b99d97f4ad9410c" exitCode=1
Jul 09 18:05:01.405369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.405340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerStarted","Data":"e219afca948545ec15e2b4a9ac78fa58a53f173a1dc193bf018ec4a53754d781"}
Jul 09 18:05:01.405422 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.405397    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerStarted","Data":"4eec76c1bf817d3b365fde5bb09361971c98a5a6cc5a8e6afd60547dbe2daa01"}
Jul 09 18:05:01.405468 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.405434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerStarted","Data":"61c83c5e1ca04bc141224c4ee1867a5a53af9de4971b4e3703420fc77eb79c78"}
Jul 09 18:05:01.405519 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.405507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerStarted","Data":"e4744fcde82ee869f3c59b37711cc927eb81c05dea53ba294eb33a1c2d777a7e"}
Jul 09 18:05:01.405555 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.405541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerDied","Data":"b158288cf19628b62e27723dad326fad4491649d81c1acd35b99d97f4ad9410c"}
Jul 09 18:05:01.405603 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.405589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerStarted","Data":"c95026c41c6cf727f821b97ed2bbfdbdd95667a42a595a6de5989b8788e87036"}
Jul 09 18:05:01.406625 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.406606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxxk7" event={"ID":"e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8","Type":"ContainerStarted","Data":"f52434051a930750112e53e8baaa1ada7a5a875a40b2ed57cb74449d635c0859"}
Jul 09 18:05:01.409969 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.409921    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal" podStartSLOduration=21.409909957 podStartE2EDuration="21.409909957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:01.409749016 +0000 UTC m=+22.736224725" watchObservedRunningTime="2026-07-09 18:05:01.409909957 +0000 UTC m=+22.736385665"
Jul 09 18:05:01.425748 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.425702    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gxxk7" podStartSLOduration=7.945715176 podStartE2EDuration="22.425690321s" totalImagesPullingTime="14.479975145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:01.42531084 +0000 UTC m=+22.751786548" watchObservedRunningTime="2026-07-09 18:05:01.425690321 +0000 UTC m=+22.752166029"
Jul 09 18:05:01.457739 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.457705    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5jhnd" podStartSLOduration=3.916852472 podStartE2EDuration="22.457692716s" totalImagesPullingTime="18.540840244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:01.439795814 +0000 UTC m=+22.766271521" watchObservedRunningTime="2026-07-09 18:05:01.457692716 +0000 UTC m=+22.784168402"
Jul 09 18:05:01.458184 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.458155    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7fb5b" podStartSLOduration=3.690798334 podStartE2EDuration="22.458148304s" totalImagesPullingTime="18.76734997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:01.45763052 +0000 UTC m=+22.784106231" watchObservedRunningTime="2026-07-09 18:05:01.458148304 +0000 UTC m=+22.784624006"
Jul 09 18:05:01.474508 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.474466    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-swk86" podStartSLOduration=3.718649916 podStartE2EDuration="22.474451286s" totalImagesPullingTime="18.75580137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:01.474155208 +0000 UTC m=+22.800630915" watchObservedRunningTime="2026-07-09 18:05:01.474451286 +0000 UTC m=+22.800926993"
Jul 09 18:05:01.516809 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.516765    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-25l7d" podStartSLOduration=3.7824575 podStartE2EDuration="22.516749355s" totalImagesPullingTime="18.734291855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:01.516236206 +0000 UTC m=+22.842711912" watchObservedRunningTime="2026-07-09 18:05:01.516749355 +0000 UTC m=+22.843225063"
Jul 09 18:05:01.990280 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:01.990257    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:05:02.246765 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:02.246637    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:05:01.990275948Z","UUID":"7dd9ecab-8d5a-43ee-939d-d16a1bd189d8","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:05:02.247846 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:02.247830    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:05:02.247932 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:02.247863    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:05:02.410779 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:02.410746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm" event={"ID":"98c0a723-9d5d-4399-a5ff-3a75d831e611","Type":"ContainerStarted","Data":"44335d03e3557eca190389b6a3f001ac549b0225bb87dce76bf70352f70816b7"}
Jul 09 18:05:02.412121 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:02.412099    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vcgq4" event={"ID":"d62dd174-274a-4a5a-99a3-069c5638ae31","Type":"ContainerStarted","Data":"00f10bff9a5cdd50482994aa2863f637f251f1843c7e1a2a1bca2d271372cfaa"}
Jul 09 18:05:02.425570 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:02.425532    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vcgq4" podStartSLOduration=4.850243414 podStartE2EDuration="23.425520266s" totalImagesPullingTime="18.575276852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:02.425031431 +0000 UTC m=+23.751507137" watchObservedRunningTime="2026-07-09 18:05:02.425520266 +0000 UTC m=+23.751995991"
Jul 09 18:05:03.309373 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:03.309163    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:03.309543 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:03.309169    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:03.309543 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:03.309463    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:05:03.309637 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:03.309574    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:05:03.309637 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:03.309256    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:03.309721 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:03.309692    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:05:03.418222 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:03.418149    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:05:03.418665 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:03.418637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerStarted","Data":"e31811ac410b9d03ce3188e817cde4d032290aa48874fdfdee674cb81e1daf6a"}
Jul 09 18:05:03.606269 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:03.606236    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:03.784733 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:03.784707    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:03.994153 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:03.994076    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:04.052216 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:04.052184    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:04.330700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:04.330619    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:04.423262 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:04.423221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm" event={"ID":"98c0a723-9d5d-4399-a5ff-3a75d831e611","Type":"ContainerStarted","Data":"e7ef8df71b2be86e297e585fe9c1629e0769e72ce46c518cafe40d762d3763df"}
Jul 09 18:05:04.464250 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:04.464225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:05.308684 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:05.308659    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:05.308780 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:05.308687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:05.308780 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:05.308659    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:05.308780 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:05.308764    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:05:05.308877 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:05.308837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:05:05.308924 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:05.308908    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:05:05.429555 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:05.429390    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:05:05.430074 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:05.429982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerStarted","Data":"467d7b1be20ffd0d7991a0cc5d106560df39e689dca3368fb1f1854260b10cda"}
Jul 09 18:05:05.430602 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:05.430584    2561 scope.go:117] "RemoveContainer" containerID="b158288cf19628b62e27723dad326fad4491649d81c1acd35b99d97f4ad9410c"
Jul 09 18:05:05.458484 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:05.458439    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w5qpm" podStartSLOduration=4.728746548 podStartE2EDuration="26.458407042s" totalImagesPullingTime="21.729660494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:04.446615594 +0000 UTC m=+25.773091313" watchObservedRunningTime="2026-07-09 18:05:05.458407042 +0000 UTC m=+26.784882751"
Jul 09 18:05:05.571599 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:05.571499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:05:05.572126 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:05.572104    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:05:06.436331 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.436305    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:05:06.436767 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.436725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" event={"ID":"08138fce-4009-438a-bf62-a17c4f40b74e","Type":"ContainerStarted","Data":"f1d566f44cf668815af545fd91166b4ce7e45d856192b4df59553d6e659c873b"}
Jul 09 18:05:06.437423 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.437291    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:05:06.437423 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.437367    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:05:06.437423 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.437398    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:05:06.437717 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.437699    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:05:06.437806 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.437776    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-25l7d"
Jul 09 18:05:06.453105 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.453083    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:05:06.454312 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.454293    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:05:06.463239 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.463195    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz" podStartSLOduration=8.470856968 podStartE2EDuration="27.463181904s" totalImagesPullingTime="18.992324936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:06.462864398 +0000 UTC m=+27.789340105" watchObservedRunningTime="2026-07-09 18:05:06.463181904 +0000 UTC m=+27.789657610"
Jul 09 18:05:06.573478 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:06.573440    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:07.285125 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:07.285097    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxrfv"]
Jul 09 18:05:07.285289 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:07.285257    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:07.285356 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:07.285338    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:05:07.288138 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:07.288113    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l5m2b"]
Jul 09 18:05:07.288259 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:07.288224    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:07.288353 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:07.288336    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:05:07.288637 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:07.288613    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fp6xc"]
Jul 09 18:05:07.288715 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:07.288702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:07.288786 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:07.288768    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:05:08.442129 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:08.441859    2561 generic.go:356] "Generic (PLEG): container finished" podID="a3555656-954a-49c5-a49c-50c44b2f7ed0" containerID="9e8099701a2434a0886e480abc5a552ef0dd7b46f66a2d3de815d0eb4c27c2c8" exitCode=0
Jul 09 18:05:08.442513 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:08.441954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zdxd" event={"ID":"a3555656-954a-49c5-a49c-50c44b2f7ed0","Type":"ContainerDied","Data":"9e8099701a2434a0886e480abc5a552ef0dd7b46f66a2d3de815d0eb4c27c2c8"}
Jul 09 18:05:09.309011 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:09.308976    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:09.309180 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:09.309078    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:09.309180 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:09.309100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:09.309319 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:09.309079    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:05:09.309319 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:09.309197    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:05:09.309319 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:09.309283    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:05:09.408828 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:09.408798    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:09.471485 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:09.471455    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:10.448140 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:10.448112    2561 generic.go:356] "Generic (PLEG): container finished" podID="a3555656-954a-49c5-a49c-50c44b2f7ed0" containerID="06aa45527ab1ca259c27382757abdacf89cdaf1ec1a8ed9478df32b803b5a5e5" exitCode=0
Jul 09 18:05:10.448289 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:10.448164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zdxd" event={"ID":"a3555656-954a-49c5-a49c-50c44b2f7ed0","Type":"ContainerDied","Data":"06aa45527ab1ca259c27382757abdacf89cdaf1ec1a8ed9478df32b803b5a5e5"}
Jul 09 18:05:11.308858 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:11.308822    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:11.308858 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:11.308853    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:11.309535 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:11.308822    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:11.309535 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:11.308925    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:05:11.309535 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:11.309010    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:05:11.309535 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:11.309143    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:05:12.453672 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:12.453641    2561 generic.go:356] "Generic (PLEG): container finished" podID="a3555656-954a-49c5-a49c-50c44b2f7ed0" containerID="7b1317ef1030a94e38686f7d2722d6ca15d33a43d80dd7aa2d2609897a5a7c07" exitCode=0
Jul 09 18:05:12.454076 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:12.453696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zdxd" event={"ID":"a3555656-954a-49c5-a49c-50c44b2f7ed0","Type":"ContainerDied","Data":"7b1317ef1030a94e38686f7d2722d6ca15d33a43d80dd7aa2d2609897a5a7c07"}
Jul 09 18:05:13.052852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.052813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:13.053045 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.052857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:13.053045 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.052987    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:05:13.053045 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.053007    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:05:13.053045 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.053040    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs podName:641493c8-16d5-46c0-bd0a-8763e005dcdf nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.053026667 +0000 UTC m=+66.379502353 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs") pod "network-metrics-daemon-l5m2b" (UID: "641493c8-16d5-46c0-bd0a-8763e005dcdf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:05:13.053287 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.053063    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret podName:5d52ad56-9937-4f10-88c3-051fe2f35027 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.053050087 +0000 UTC m=+66.379525772 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret") pod "global-pull-secret-syncer-fxrfv" (UID: "5d52ad56-9937-4f10-88c3-051fe2f35027") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:05:13.153831 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.153800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:13.154026 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.153960    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:05:13.154026 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.153984    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:05:13.154026 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.153996    2561 projected.go:196] Error preparing data for projected volume kube-api-access-wgxgn for pod openshift-network-diagnostics/network-check-target-fp6xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:05:13.154140 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.154048    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn podName:1f662f2f-4128-4b2d-be2a-e3053a6f47e5 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.154031919 +0000 UTC m=+66.480507626 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wgxgn" (UniqueName: "kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn") pod "network-check-target-fp6xc" (UID: "1f662f2f-4128-4b2d-be2a-e3053a6f47e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:05:13.308467 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.308394    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:13.308624 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.308405    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:13.308624 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.308519    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fxrfv" podUID="5d52ad56-9937-4f10-88c3-051fe2f35027"
Jul 09 18:05:13.308624 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.308402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:13.308624 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.308608    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l5m2b" podUID="641493c8-16d5-46c0-bd0a-8763e005dcdf"
Jul 09 18:05:13.308793 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:13.308664    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fp6xc" podUID="1f662f2f-4128-4b2d-be2a-e3053a6f47e5"
Jul 09 18:05:13.521231 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.521180    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:13.573303 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.573221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:13.825747 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.825673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:13.916291 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.916266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:13.989205 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:13.989164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:14.020735 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.020712    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeReady"
Jul 09 18:05:14.020848 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.020835    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:05:14.061637 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.061607    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d8b87f76-79hcx"]
Jul 09 18:05:14.080882 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.080791    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9f5n4"]
Jul 09 18:05:14.081013 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.080912    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.083467 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.083443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:05:14.083598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.083450    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kz2r6\""
Jul 09 18:05:14.083598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.083567    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:05:14.083789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.083767    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:05:14.090100 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.090078    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:05:14.092994 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.092974    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wg2jq"]
Jul 09 18:05:14.093101 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.093087    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.095678 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.095656    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:05:14.095837 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.095823    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:05:14.095891 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.095872    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2sdwn\""
Jul 09 18:05:14.113158 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.113133    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8b87f76-79hcx"]
Jul 09 18:05:14.113158 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.113159    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wg2jq"]
Jul 09 18:05:14.113313 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.113171    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9f5n4"]
Jul 09 18:05:14.113353 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.113315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wg2jq"
Jul 09 18:05:14.115623 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.115604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:05:14.115726 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.115649    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:05:14.115726 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.115680    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rtw6h\""
Jul 09 18:05:14.115834 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.115753    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:05:14.195936 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.195899    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:14.261687 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-bound-sa-token\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.261889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-config-volume\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.261889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.261889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-registry-certificates\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.261889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.261889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-installation-pull-secrets\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.261889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvrf\" (UniqueName: \"kubernetes.io/projected/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-kube-api-access-9fvrf\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.261889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6rq\" (UniqueName: \"kubernetes.io/projected/9c23fe5b-f395-4f2d-9304-52e9d6adce51-kube-api-access-sj6rq\") pod \"ingress-canary-wg2jq\" (UID: \"9c23fe5b-f395-4f2d-9304-52e9d6adce51\") " pod="openshift-ingress-canary/ingress-canary-wg2jq"
Jul 09 18:05:14.261889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-image-registry-private-configuration\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.261889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-tmp-dir\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.262336 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.261988    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43cceb62-245b-40a5-b660-c826730cf06a-ca-trust-extracted\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.262336 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.262008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c23fe5b-f395-4f2d-9304-52e9d6adce51-cert\") pod \"ingress-canary-wg2jq\" (UID: \"9c23fe5b-f395-4f2d-9304-52e9d6adce51\") " pod="openshift-ingress-canary/ingress-canary-wg2jq"
Jul 09 18:05:14.262336 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.262030    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-trusted-ca\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.262336 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.262054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc94g\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-kube-api-access-lc94g\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.363239 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-bound-sa-token\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.363239 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-config-volume\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.363239 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.363498 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-registry-certificates\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.363498 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.363498 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-installation-pull-secrets\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.363498 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvrf\" (UniqueName: \"kubernetes.io/projected/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-kube-api-access-9fvrf\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.363498 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.363342    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:14.363498 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.363364    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8b87f76-79hcx: secret "image-registry-tls" not found
Jul 09 18:05:14.363498 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.363403    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:14.363498 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.363432    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls podName:43cceb62-245b-40a5-b660-c826730cf06a nodeName:}" failed. No retries permitted until 2026-07-09 18:05:14.863412229 +0000 UTC m=+36.189887935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls") pod "image-registry-5d8b87f76-79hcx" (UID: "43cceb62-245b-40a5-b660-c826730cf06a") : secret "image-registry-tls" not found
Jul 09 18:05:14.363498 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.363486    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls podName:d5ccc282-8bfa-47dc-80b1-13c7232c51ab nodeName:}" failed. No retries permitted until 2026-07-09 18:05:14.863473337 +0000 UTC m=+36.189949043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls") pod "dns-default-9f5n4" (UID: "d5ccc282-8bfa-47dc-80b1-13c7232c51ab") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:14.363797 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6rq\" (UniqueName: \"kubernetes.io/projected/9c23fe5b-f395-4f2d-9304-52e9d6adce51-kube-api-access-sj6rq\") pod \"ingress-canary-wg2jq\" (UID: \"9c23fe5b-f395-4f2d-9304-52e9d6adce51\") " pod="openshift-ingress-canary/ingress-canary-wg2jq"
Jul 09 18:05:14.363797 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-image-registry-private-configuration\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.363797 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-tmp-dir\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.363797 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-config-volume\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.363970 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43cceb62-245b-40a5-b660-c826730cf06a-ca-trust-extracted\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.363970 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c23fe5b-f395-4f2d-9304-52e9d6adce51-cert\") pod \"ingress-canary-wg2jq\" (UID: \"9c23fe5b-f395-4f2d-9304-52e9d6adce51\") " pod="openshift-ingress-canary/ingress-canary-wg2jq"
Jul 09 18:05:14.363970 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363895    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-trusted-ca\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.363970 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.363918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc94g\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-kube-api-access-lc94g\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.364167 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.364121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-registry-certificates\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.364167 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.364143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-tmp-dir\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.364685 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.364593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43cceb62-245b-40a5-b660-c826730cf06a-ca-trust-extracted\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.364799 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.364777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-trusted-ca\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.367471 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.367446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-image-registry-private-configuration\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.367548 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.367448    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-installation-pull-secrets\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.370286 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.370266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c23fe5b-f395-4f2d-9304-52e9d6adce51-cert\") pod \"ingress-canary-wg2jq\" (UID: \"9c23fe5b-f395-4f2d-9304-52e9d6adce51\") " pod="openshift-ingress-canary/ingress-canary-wg2jq"
Jul 09 18:05:14.386814 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.386795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6rq\" (UniqueName: \"kubernetes.io/projected/9c23fe5b-f395-4f2d-9304-52e9d6adce51-kube-api-access-sj6rq\") pod \"ingress-canary-wg2jq\" (UID: \"9c23fe5b-f395-4f2d-9304-52e9d6adce51\") " pod="openshift-ingress-canary/ingress-canary-wg2jq"
Jul 09 18:05:14.388170 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.388145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-bound-sa-token\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.393569 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.393545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc94g\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-kube-api-access-lc94g\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.395680 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.395659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvrf\" (UniqueName: \"kubernetes.io/projected/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-kube-api-access-9fvrf\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.423099 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.423071    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wg2jq"
Jul 09 18:05:14.538137 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.538095    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gx8b2"]
Jul 09 18:05:14.550904 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.550869    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.551982 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.551900    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gx8b2"]
Jul 09 18:05:14.553224 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.553165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:05:14.553587 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.553562    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:05:14.553803 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.553787    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:05:14.553862 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.553835    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qd2fl\""
Jul 09 18:05:14.554449 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.554227    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:05:14.573087 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.573059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:14.582750 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.582727    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wg2jq"]
Jul 09 18:05:14.587661 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:14.587635    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c23fe5b_f395_4f2d_9304_52e9d6adce51.slice/crio-1dc22d700bdbee875c39d4dee2856d3fc28c61a281af66f6de21938829cc67d5 WatchSource:0}: Error finding container 1dc22d700bdbee875c39d4dee2856d3fc28c61a281af66f6de21938829cc67d5: Status 404 returned error can't find the container with id 1dc22d700bdbee875c39d4dee2856d3fc28c61a281af66f6de21938829cc67d5
Jul 09 18:05:14.666658 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.666585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbn9s\" (UniqueName: \"kubernetes.io/projected/03a22790-e419-40d7-abdb-6517d9103e1c-kube-api-access-jbn9s\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.666658 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.666624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.666862 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.666701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03a22790-e419-40d7-abdb-6517d9103e1c-crio-socket\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.666862 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.666772    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03a22790-e419-40d7-abdb-6517d9103e1c-data-volume\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.666862 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.666793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.767654 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.767623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03a22790-e419-40d7-abdb-6517d9103e1c-data-volume\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.767835 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.767667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.767835 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.767751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbn9s\" (UniqueName: \"kubernetes.io/projected/03a22790-e419-40d7-abdb-6517d9103e1c-kube-api-access-jbn9s\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.767835 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.767781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.768052 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.767837    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:14.768052 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.767911    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:14.768052 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.767916    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm podName:03a22790-e419-40d7-abdb-6517d9103e1c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.267894492 +0000 UTC m=+36.594370198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gx8b2" (UID: "03a22790-e419-40d7-abdb-6517d9103e1c") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:14.768052 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.767982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03a22790-e419-40d7-abdb-6517d9103e1c-crio-socket\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.768052 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.767994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03a22790-e419-40d7-abdb-6517d9103e1c-data-volume\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.768052 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.768032    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls podName:03a22790-e419-40d7-abdb-6517d9103e1c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.268019333 +0000 UTC m=+36.594495022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gx8b2" (UID: "03a22790-e419-40d7-abdb-6517d9103e1c") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:14.768289 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.768121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03a22790-e419-40d7-abdb-6517d9103e1c-crio-socket\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.776958 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.776923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbn9s\" (UniqueName: \"kubernetes.io/projected/03a22790-e419-40d7-abdb-6517d9103e1c-kube-api-access-jbn9s\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:14.819580 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.819556    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:14.868801 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.868777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:14.868981 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:14.868806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:14.868981 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.868928    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:14.868981 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.868930    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:14.868981 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.868963    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8b87f76-79hcx: secret "image-registry-tls" not found
Jul 09 18:05:14.869147 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.868994    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls podName:d5ccc282-8bfa-47dc-80b1-13c7232c51ab nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.868977032 +0000 UTC m=+37.195452718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls") pod "dns-default-9f5n4" (UID: "d5ccc282-8bfa-47dc-80b1-13c7232c51ab") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:14.869147 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:14.869022    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls podName:43cceb62-245b-40a5-b660-c826730cf06a nodeName:}" failed. No retries permitted until 2026-07-09 18:05:15.869004549 +0000 UTC m=+37.195480241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls") pod "image-registry-5d8b87f76-79hcx" (UID: "43cceb62-245b-40a5-b660-c826730cf06a") : secret "image-registry-tls" not found
Jul 09 18:05:15.218146 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.218100    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.272245 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.272215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:15.272412 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.272294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:15.272517 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:15.272495    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:15.272643 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:15.272563    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm podName:03a22790-e419-40d7-abdb-6517d9103e1c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:16.272544181 +0000 UTC m=+37.599019882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gx8b2" (UID: "03a22790-e419-40d7-abdb-6517d9103e1c") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:15.272874 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:15.272788    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:15.272874 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:15.272853    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls podName:03a22790-e419-40d7-abdb-6517d9103e1c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:16.272838464 +0000 UTC m=+37.599314150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gx8b2" (UID: "03a22790-e419-40d7-abdb-6517d9103e1c") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:15.308886 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.308859    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:15.309092 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.309071    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:15.309404 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.308862    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:15.311698 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.311667    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pkf52\""
Jul 09 18:05:15.311698 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.311678    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6mch4\""
Jul 09 18:05:15.311850 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.311678    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:15.312508 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.312488    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:15.312702 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.312688    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:15.312870 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.312852    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:15.322367 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.322344    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.462293 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.462202    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wg2jq" event={"ID":"9c23fe5b-f395-4f2d-9304-52e9d6adce51","Type":"ContainerStarted","Data":"1dc22d700bdbee875c39d4dee2856d3fc28c61a281af66f6de21938829cc67d5"}
Jul 09 18:05:15.498665 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.498588    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.606785 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.606747    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.878931 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.878851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:15.878931 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:15.878896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:15.879166 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:15.879035    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:15.879166 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:15.879054    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8b87f76-79hcx: secret "image-registry-tls" not found
Jul 09 18:05:15.879166 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:15.879110    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:15.879166 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:15.879121    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls podName:43cceb62-245b-40a5-b660-c826730cf06a nodeName:}" failed. No retries permitted until 2026-07-09 18:05:17.879100026 +0000 UTC m=+39.205575733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls") pod "image-registry-5d8b87f76-79hcx" (UID: "43cceb62-245b-40a5-b660-c826730cf06a") : secret "image-registry-tls" not found
Jul 09 18:05:15.879166 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:15.879157    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls podName:d5ccc282-8bfa-47dc-80b1-13c7232c51ab nodeName:}" failed. No retries permitted until 2026-07-09 18:05:17.879142921 +0000 UTC m=+39.205618613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls") pod "dns-default-9f5n4" (UID: "d5ccc282-8bfa-47dc-80b1-13c7232c51ab") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:16.283396 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:16.283361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:16.283582 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:16.283432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:16.283582 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:16.283510    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:16.283582 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:16.283519    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:16.283582 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:16.283576    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm podName:03a22790-e419-40d7-abdb-6517d9103e1c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:18.28355807 +0000 UTC m=+39.610033759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gx8b2" (UID: "03a22790-e419-40d7-abdb-6517d9103e1c") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:16.283801 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:16.283593    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls podName:03a22790-e419-40d7-abdb-6517d9103e1c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:18.283584856 +0000 UTC m=+39.610060541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gx8b2" (UID: "03a22790-e419-40d7-abdb-6517d9103e1c") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:17.018433 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.018393    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxxk7_e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8/dns-node-resolver/0.log"
Jul 09 18:05:17.084642 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.084567    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xs6nl"]
Jul 09 18:05:17.099360 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.099326    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xs6nl"]
Jul 09 18:05:17.099516 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.099455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.102112 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.102075    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 09 18:05:17.102112 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.102089    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 09 18:05:17.103102 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.103078    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 09 18:05:17.103219 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.103081    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lqzkk\""
Jul 09 18:05:17.103219 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.103142    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 09 18:05:17.291070 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.291033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03b50e87-ce84-4dc7-abff-978c49834fda-signing-cabundle\") pod \"service-ca-79f68b9f4f-xs6nl\" (UID: \"03b50e87-ce84-4dc7-abff-978c49834fda\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.291247 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.291093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03b50e87-ce84-4dc7-abff-978c49834fda-signing-key\") pod \"service-ca-79f68b9f4f-xs6nl\" (UID: \"03b50e87-ce84-4dc7-abff-978c49834fda\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.291247 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.291220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8n7\" (UniqueName: \"kubernetes.io/projected/03b50e87-ce84-4dc7-abff-978c49834fda-kube-api-access-5q8n7\") pod \"service-ca-79f68b9f4f-xs6nl\" (UID: \"03b50e87-ce84-4dc7-abff-978c49834fda\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.392327 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.392250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8n7\" (UniqueName: \"kubernetes.io/projected/03b50e87-ce84-4dc7-abff-978c49834fda-kube-api-access-5q8n7\") pod \"service-ca-79f68b9f4f-xs6nl\" (UID: \"03b50e87-ce84-4dc7-abff-978c49834fda\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.392495 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.392349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03b50e87-ce84-4dc7-abff-978c49834fda-signing-cabundle\") pod \"service-ca-79f68b9f4f-xs6nl\" (UID: \"03b50e87-ce84-4dc7-abff-978c49834fda\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.392495 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.392396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03b50e87-ce84-4dc7-abff-978c49834fda-signing-key\") pod \"service-ca-79f68b9f4f-xs6nl\" (UID: \"03b50e87-ce84-4dc7-abff-978c49834fda\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.393112 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.393087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/03b50e87-ce84-4dc7-abff-978c49834fda-signing-cabundle\") pod \"service-ca-79f68b9f4f-xs6nl\" (UID: \"03b50e87-ce84-4dc7-abff-978c49834fda\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.396081 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.395892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/03b50e87-ce84-4dc7-abff-978c49834fda-signing-key\") pod \"service-ca-79f68b9f4f-xs6nl\" (UID: \"03b50e87-ce84-4dc7-abff-978c49834fda\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.403361 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.403339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8n7\" (UniqueName: \"kubernetes.io/projected/03b50e87-ce84-4dc7-abff-978c49834fda-kube-api-access-5q8n7\") pod \"service-ca-79f68b9f4f-xs6nl\" (UID: \"03b50e87-ce84-4dc7-abff-978c49834fda\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.409267 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.409227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl"
Jul 09 18:05:17.476099 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.476072    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.536404 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.536369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.818772 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.818746    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5jhnd_9ec2f0e2-e620-4920-8313-1a3107dca62a/node-ca/0.log"
Jul 09 18:05:17.895687 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.895657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:17.895687 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:17.895689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:17.895872 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:17.895803    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:17.895872 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:17.895810    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:17.895872 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:17.895829    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8b87f76-79hcx: secret "image-registry-tls" not found
Jul 09 18:05:17.895872 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:17.895851    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls podName:d5ccc282-8bfa-47dc-80b1-13c7232c51ab nodeName:}" failed. No retries permitted until 2026-07-09 18:05:21.895838383 +0000 UTC m=+43.222314069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls") pod "dns-default-9f5n4" (UID: "d5ccc282-8bfa-47dc-80b1-13c7232c51ab") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:17.895872 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:17.895870    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls podName:43cceb62-245b-40a5-b660-c826730cf06a nodeName:}" failed. No retries permitted until 2026-07-09 18:05:21.89585656 +0000 UTC m=+43.222332245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls") pod "image-registry-5d8b87f76-79hcx" (UID: "43cceb62-245b-40a5-b660-c826730cf06a") : secret "image-registry-tls" not found
Jul 09 18:05:18.012442 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:18.012412    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:18.284021 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:18.283993    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:18.298196 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:18.298173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:18.298283 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:18.298228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:18.298325 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:18.298285    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:18.298325 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:18.298315    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:18.298325 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:18.298326    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm podName:03a22790-e419-40d7-abdb-6517d9103e1c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:22.298314858 +0000 UTC m=+43.624790544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gx8b2" (UID: "03a22790-e419-40d7-abdb-6517d9103e1c") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:18.298442 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:18.298364    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls podName:03a22790-e419-40d7-abdb-6517d9103e1c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:22.298350408 +0000 UTC m=+43.624826094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gx8b2" (UID: "03a22790-e419-40d7-abdb-6517d9103e1c") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:18.387289 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:18.387209    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:18.487580 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:18.487315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xs6nl"]
Jul 09 18:05:18.492404 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:18.492368    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b50e87_ce84_4dc7_abff_978c49834fda.slice/crio-187151c19fc46db22a0d58806c8161be1e6194ddb5101abb8336b213fedbb2f2 WatchSource:0}: Error finding container 187151c19fc46db22a0d58806c8161be1e6194ddb5101abb8336b213fedbb2f2: Status 404 returned error can't find the container with id 187151c19fc46db22a0d58806c8161be1e6194ddb5101abb8336b213fedbb2f2
Jul 09 18:05:18.526142 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:18.526117    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:19.478061 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:19.478026    2561 generic.go:356] "Generic (PLEG): container finished" podID="a3555656-954a-49c5-a49c-50c44b2f7ed0" containerID="43a56556b01c8a85a56b0138df4a525baf6f7a1c329f69172e337c774184faca" exitCode=0
Jul 09 18:05:19.478599 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:19.478104    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zdxd" event={"ID":"a3555656-954a-49c5-a49c-50c44b2f7ed0","Type":"ContainerDied","Data":"43a56556b01c8a85a56b0138df4a525baf6f7a1c329f69172e337c774184faca"}
Jul 09 18:05:19.479180 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:19.479158    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl" event={"ID":"03b50e87-ce84-4dc7-abff-978c49834fda","Type":"ContainerStarted","Data":"187151c19fc46db22a0d58806c8161be1e6194ddb5101abb8336b213fedbb2f2"}
Jul 09 18:05:19.480494 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:19.480476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wg2jq" event={"ID":"9c23fe5b-f395-4f2d-9304-52e9d6adce51","Type":"ContainerStarted","Data":"5aa13e7efc0654d1693f7d173b9a6605974a5d3127eb3335f0da1129c4293918"}
Jul 09 18:05:20.274456 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:20.274422    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:20.486350 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:20.486318    2561 generic.go:356] "Generic (PLEG): container finished" podID="a3555656-954a-49c5-a49c-50c44b2f7ed0" containerID="1a4527be119c7d903f0e48f3464217a9cbf8b03b8efd0249988448bd81a8bb25" exitCode=0
Jul 09 18:05:20.486800 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:20.486418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zdxd" event={"ID":"a3555656-954a-49c5-a49c-50c44b2f7ed0","Type":"ContainerDied","Data":"1a4527be119c7d903f0e48f3464217a9cbf8b03b8efd0249988448bd81a8bb25"}
Jul 09 18:05:20.512187 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:20.512141    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wg2jq" podStartSLOduration=2.760395751 podStartE2EDuration="6.512120723s" totalImagesPullingTime="3.751724972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:19.514502001 +0000 UTC m=+40.840977726" watchObservedRunningTime="2026-07-09 18:05:20.512120723 +0000 UTC m=+41.838596442"
Jul 09 18:05:21.493080 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:21.493043    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zdxd" event={"ID":"a3555656-954a-49c5-a49c-50c44b2f7ed0","Type":"ContainerStarted","Data":"504f147c67abcd16e17294cb3845d07c4efa99e6b34262cedafff7d8af3d8b15"}
Jul 09 18:05:21.522539 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:21.522488    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4zdxd" podStartSLOduration=9.372026919 podStartE2EDuration="42.52246995s" totalImagesPullingTime="33.150443031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:21.521830416 +0000 UTC m=+42.848306140" watchObservedRunningTime="2026-07-09 18:05:21.52246995 +0000 UTC m=+42.848945657"
Jul 09 18:05:21.869789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:21.869699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:21.931404 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:21.931367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:21.931404 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:21.931404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:21.931586 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:21.931511    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:05:21.931586 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:21.931533    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8b87f76-79hcx: secret "image-registry-tls" not found
Jul 09 18:05:21.931586 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:21.931584    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls podName:43cceb62-245b-40a5-b660-c826730cf06a nodeName:}" failed. No retries permitted until 2026-07-09 18:05:29.931566464 +0000 UTC m=+51.258042151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls") pod "image-registry-5d8b87f76-79hcx" (UID: "43cceb62-245b-40a5-b660-c826730cf06a") : secret "image-registry-tls" not found
Jul 09 18:05:21.931722 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:21.931516    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:21.931722 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:21.931673    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls podName:d5ccc282-8bfa-47dc-80b1-13c7232c51ab nodeName:}" failed. No retries permitted until 2026-07-09 18:05:29.931651384 +0000 UTC m=+51.258127073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls") pod "dns-default-9f5n4" (UID: "d5ccc282-8bfa-47dc-80b1-13c7232c51ab") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:22.334176 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:22.334149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:22.334291 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:22.334232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:22.334356 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:22.334343    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:22.334420 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:22.334408    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls podName:03a22790-e419-40d7-abdb-6517d9103e1c nodeName:}" failed. No retries permitted until 2026-07-09 18:05:30.334390475 +0000 UTC m=+51.660866184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gx8b2" (UID: "03a22790-e419-40d7-abdb-6517d9103e1c") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:22.334718 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:22.334698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a22790-e419-40d7-abdb-6517d9103e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:22.496702 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:22.496666    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl" event={"ID":"03b50e87-ce84-4dc7-abff-978c49834fda","Type":"ContainerStarted","Data":"d8e66a6016b9823d5cc851a1799bf7cc59f3e45db1277957eb94d7f26cc18816"}
Jul 09 18:05:22.515425 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:22.515384    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-xs6nl" podStartSLOduration=1.67541812 podStartE2EDuration="5.515370427s" totalImagesPullingTime="3.839952307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:22.514551125 +0000 UTC m=+43.841026832" watchObservedRunningTime="2026-07-09 18:05:22.515370427 +0000 UTC m=+43.841846134"
Jul 09 18:05:23.793403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:23.793369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:23.896010 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:23.895964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:24.070361 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:24.070274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:24.130628 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:24.130595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:24.236343 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:24.236312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:24.482576 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:24.482546    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:25.486590 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:25.486562    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:25.886627 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:25.886554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:26.079656 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:26.079620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:26.285473 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:26.285445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:26.488320 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:26.488291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:27.409709 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:27.409679    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:27.510584 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:27.510555    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:27.570545 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:27.570518    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:27.686418 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:27.686383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.102271 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:28.102172    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.284101 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:28.284073    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.347100 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:28.347073    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.705733 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:28.705701    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.289608 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:29.289578    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.390462 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:29.390387    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.490889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:29.490863    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.886352 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:29.886313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.992324 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:29.992290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:29.992458 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:29.992332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:29.994765 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:29.994739    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ccc282-8bfa-47dc-80b1-13c7232c51ab-metrics-tls\") pod \"dns-default-9f5n4\" (UID: \"d5ccc282-8bfa-47dc-80b1-13c7232c51ab\") " pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:29.994886 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:29.994863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") pod \"image-registry-5d8b87f76-79hcx\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") " pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:30.004723 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.004702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:30.096604 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.096561    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:30.130611 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.130582    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9f5n4"]
Jul 09 18:05:30.134182 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:30.134156    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ccc282_8bfa_47dc_80b1_13c7232c51ab.slice/crio-295ed6a938de10ace96b1cb3d167c38896929e9de25ddf2983c19cd559c5d77f WatchSource:0}: Error finding container 295ed6a938de10ace96b1cb3d167c38896929e9de25ddf2983c19cd559c5d77f: Status 404 returned error can't find the container with id 295ed6a938de10ace96b1cb3d167c38896929e9de25ddf2983c19cd559c5d77f
Jul 09 18:05:30.170867 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.170840    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:30.293895 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.293866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:30.397599 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.397527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:30.399889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.399864    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a22790-e419-40d7-abdb-6517d9103e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gx8b2\" (UID: \"03a22790-e419-40d7-abdb-6517d9103e1c\") " pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:30.422352 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.422327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8b87f76-79hcx"]
Jul 09 18:05:30.425153 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:30.425122    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cceb62_245b_40a5_b660_c826730cf06a.slice/crio-551836d4f587dfdd71352ef09382e1479affddde17ad363eae2b41c7c2c2c46b WatchSource:0}: Error finding container 551836d4f587dfdd71352ef09382e1479affddde17ad363eae2b41c7c2c2c46b: Status 404 returned error can't find the container with id 551836d4f587dfdd71352ef09382e1479affddde17ad363eae2b41c7c2c2c46b
Jul 09 18:05:30.462905 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.462887    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gx8b2"
Jul 09 18:05:30.516038 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.515982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9f5n4" event={"ID":"d5ccc282-8bfa-47dc-80b1-13c7232c51ab","Type":"ContainerStarted","Data":"295ed6a938de10ace96b1cb3d167c38896929e9de25ddf2983c19cd559c5d77f"}
Jul 09 18:05:30.517856 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.517825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx" event={"ID":"43cceb62-245b-40a5-b660-c826730cf06a","Type":"ContainerStarted","Data":"b3d6e05efdce93c88e949f2cca7e32e74aefa530c80f86258fb63956516a3fd6"}
Jul 09 18:05:30.517982 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.517877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx" event={"ID":"43cceb62-245b-40a5-b660-c826730cf06a","Type":"ContainerStarted","Data":"551836d4f587dfdd71352ef09382e1479affddde17ad363eae2b41c7c2c2c46b"}
Jul 09 18:05:30.518218 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.518159    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:30.540985 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.540920    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx" podStartSLOduration=47.540902395 podStartE2EDuration="47.540902395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:30.540213443 +0000 UTC m=+51.866689161" watchObservedRunningTime="2026-07-09 18:05:30.540902395 +0000 UTC m=+51.867378100"
Jul 09 18:05:30.591633 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:30.591601    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gx8b2"]
Jul 09 18:05:30.596776 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:30.596748    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a22790_e419_40d7_abdb_6517d9103e1c.slice/crio-8e801209d761afde037d97e94dc7e463e233e218509235e6dc568d3bcf3ddea8 WatchSource:0}: Error finding container 8e801209d761afde037d97e94dc7e463e233e218509235e6dc568d3bcf3ddea8: Status 404 returned error can't find the container with id 8e801209d761afde037d97e94dc7e463e233e218509235e6dc568d3bcf3ddea8
Jul 09 18:05:31.190098 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:31.190016    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:31.307794 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:31.307761    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:31.523675 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:31.523552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gx8b2" event={"ID":"03a22790-e419-40d7-abdb-6517d9103e1c","Type":"ContainerStarted","Data":"8e801209d761afde037d97e94dc7e463e233e218509235e6dc568d3bcf3ddea8"}
Jul 09 18:05:31.895525 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:31.895428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:32.092142 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:32.092067    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:32.484659 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:32.484603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:32.529284 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:32.529248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9f5n4" event={"ID":"d5ccc282-8bfa-47dc-80b1-13c7232c51ab","Type":"ContainerStarted","Data":"38d6a937453b877a06d3e7f5cb674e480a48c0b037530035e85c723ac322250e"}
Jul 09 18:05:32.529455 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:32.529331    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9f5n4" event={"ID":"d5ccc282-8bfa-47dc-80b1-13c7232c51ab","Type":"ContainerStarted","Data":"a57ec5ca68e5c226d394d6ff6bec096fb8089b8813253d7af3794bf9211059b0"}
Jul 09 18:05:32.529455 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:32.529380    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:32.547349 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:32.547292    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9f5n4" podStartSLOduration=16.645164852 podStartE2EDuration="18.547278744s" totalImagesPullingTime="1.902113892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:32.546310579 +0000 UTC m=+53.872786287" watchObservedRunningTime="2026-07-09 18:05:32.547278744 +0000 UTC m=+53.873754451"
Jul 09 18:05:32.681104 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:32.681063    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:33.096962 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:33.096907    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:33.277278 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:33.277234    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:33.378401 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:33.378310    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:33.685115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:33.685033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:34.087625 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:34.087593    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:34.282603 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:34.282565    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:34.494784 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:34.494750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:35.539370 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:35.539330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gx8b2" event={"ID":"03a22790-e419-40d7-abdb-6517d9103e1c","Type":"ContainerStarted","Data":"a1ca46d31a3e00af2bd0cabe18910e60e4bd4b3abb476a63bbbaa880d8e02219"}
Jul 09 18:05:36.548427 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:36.548384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gx8b2" event={"ID":"03a22790-e419-40d7-abdb-6517d9103e1c","Type":"ContainerStarted","Data":"4eaa846f013837bb37b02ec7000b9619f9c4cfbf62495372a1c2c21a835f9307"}
Jul 09 18:05:37.596119 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:37.596082    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.692340 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:37.692299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.760120 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:37.760087    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.864628 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:37.864555    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.050244 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.050211    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.150824 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.150745    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.367496 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.367466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.452285 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.452263    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8jhz"
Jul 09 18:05:38.463778 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.463754    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d8b87f76-79hcx"]
Jul 09 18:05:38.471835 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.471814    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"]
Jul 09 18:05:38.475258 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.475240    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"]
Jul 09 18:05:38.475376 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.475358    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.478639 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.478618    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 18:05:38.479251 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.478939    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"
Jul 09 18:05:38.479251 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.479141    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 18:05:38.479251 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.479153    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 18:05:38.479251 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.479229    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.479673 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.479656    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 18:05:38.480088 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.480073    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"]
Jul 09 18:05:38.482409 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.482385    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 18:05:38.482545 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.482467    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mbmgx\""
Jul 09 18:05:38.484351 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.484336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.486302 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.486283    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"]
Jul 09 18:05:38.487005 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.486986    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 18:05:38.487166 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.487033    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 18:05:38.487256 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.487049    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 18:05:38.487525 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.487512    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 18:05:38.489347 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.489330    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"]
Jul 09 18:05:38.522771 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.522752    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"]
Jul 09 18:05:38.567687 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.567660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35707bf-0576-4f74-9306-2fe4bd43d35e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.567770 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.567694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.567770 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.567724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ffd29531-ff87-48c4-8cd2-8f309ff50e6c-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cdf68bf66-djbjw\" (UID: \"ffd29531-ff87-48c4-8cd2-8f309ff50e6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.567770 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.567749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30b8ee01-7117-4361-a178-ea00145e6765-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56b5665ffd-f7488\" (UID: \"30b8ee01-7117-4361-a178-ea00145e6765\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"
Jul 09 18:05:38.567869 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.567779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd29531-ff87-48c4-8cd2-8f309ff50e6c-tmp\") pod \"klusterlet-addon-workmgr-6cdf68bf66-djbjw\" (UID: \"ffd29531-ff87-48c4-8cd2-8f309ff50e6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.567869 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.567844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqst8\" (UniqueName: \"kubernetes.io/projected/30b8ee01-7117-4361-a178-ea00145e6765-kube-api-access-dqst8\") pod \"managed-serviceaccount-addon-agent-56b5665ffd-f7488\" (UID: \"30b8ee01-7117-4361-a178-ea00145e6765\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"
Jul 09 18:05:38.567954 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.567886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-ca\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.567954 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.567902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l77bx\" (UniqueName: \"kubernetes.io/projected/f35707bf-0576-4f74-9306-2fe4bd43d35e-kube-api-access-l77bx\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.568073 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.568057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g59kv\" (UniqueName: \"kubernetes.io/projected/ffd29531-ff87-48c4-8cd2-8f309ff50e6c-kube-api-access-g59kv\") pod \"klusterlet-addon-workmgr-6cdf68bf66-djbjw\" (UID: \"ffd29531-ff87-48c4-8cd2-8f309ff50e6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.568150 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.568122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-hub\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.568211 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.568164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.578972 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.578936    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6666f97c54-7dng9"]
Jul 09 18:05:38.581797 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.581783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.628258 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.628226    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.629174 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.629157    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6666f97c54-7dng9"]
Jul 09 18:05:38.668727 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.668705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqst8\" (UniqueName: \"kubernetes.io/projected/30b8ee01-7117-4361-a178-ea00145e6765-kube-api-access-dqst8\") pod \"managed-serviceaccount-addon-agent-56b5665ffd-f7488\" (UID: \"30b8ee01-7117-4361-a178-ea00145e6765\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"
Jul 09 18:05:38.668820 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.668740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71500497-abe2-44b4-afd0-e956f18e2d01-registry-tls\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.668820 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.668763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-ca\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.668820 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.668786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l77bx\" (UniqueName: \"kubernetes.io/projected/f35707bf-0576-4f74-9306-2fe4bd43d35e-kube-api-access-l77bx\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.668820 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.668811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71500497-abe2-44b4-afd0-e956f18e2d01-installation-pull-secrets\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.668985 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.668836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71500497-abe2-44b4-afd0-e956f18e2d01-registry-certificates\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.668985 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.668861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g59kv\" (UniqueName: \"kubernetes.io/projected/ffd29531-ff87-48c4-8cd2-8f309ff50e6c-kube-api-access-g59kv\") pod \"klusterlet-addon-workmgr-6cdf68bf66-djbjw\" (UID: \"ffd29531-ff87-48c4-8cd2-8f309ff50e6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.668985 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.668886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71500497-abe2-44b4-afd0-e956f18e2d01-bound-sa-token\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.668985 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.668971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-hub\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.669171 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71500497-abe2-44b4-afd0-e956f18e2d01-image-registry-private-configuration\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.669171 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.669171 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71500497-abe2-44b4-afd0-e956f18e2d01-ca-trust-extracted\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.669171 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35707bf-0576-4f74-9306-2fe4bd43d35e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.669171 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxmsm\" (UniqueName: \"kubernetes.io/projected/71500497-abe2-44b4-afd0-e956f18e2d01-kube-api-access-dxmsm\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.669415 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.669415 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71500497-abe2-44b4-afd0-e956f18e2d01-trusted-ca\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.669415 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ffd29531-ff87-48c4-8cd2-8f309ff50e6c-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cdf68bf66-djbjw\" (UID: \"ffd29531-ff87-48c4-8cd2-8f309ff50e6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.669558 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30b8ee01-7117-4361-a178-ea00145e6765-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56b5665ffd-f7488\" (UID: \"30b8ee01-7117-4361-a178-ea00145e6765\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"
Jul 09 18:05:38.669558 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd29531-ff87-48c4-8cd2-8f309ff50e6c-tmp\") pod \"klusterlet-addon-workmgr-6cdf68bf66-djbjw\" (UID: \"ffd29531-ff87-48c4-8cd2-8f309ff50e6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.669826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.669802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35707bf-0576-4f74-9306-2fe4bd43d35e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.670528 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.670502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd29531-ff87-48c4-8cd2-8f309ff50e6c-tmp\") pod \"klusterlet-addon-workmgr-6cdf68bf66-djbjw\" (UID: \"ffd29531-ff87-48c4-8cd2-8f309ff50e6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.671794 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.671771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-ca\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.671939 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.671914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.672107 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.672086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-hub\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.672166 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.672103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30b8ee01-7117-4361-a178-ea00145e6765-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56b5665ffd-f7488\" (UID: \"30b8ee01-7117-4361-a178-ea00145e6765\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"
Jul 09 18:05:38.672166 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.672092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ffd29531-ff87-48c4-8cd2-8f309ff50e6c-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cdf68bf66-djbjw\" (UID: \"ffd29531-ff87-48c4-8cd2-8f309ff50e6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.672166 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.672151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f35707bf-0576-4f74-9306-2fe4bd43d35e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.688243 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.688217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g59kv\" (UniqueName: \"kubernetes.io/projected/ffd29531-ff87-48c4-8cd2-8f309ff50e6c-kube-api-access-g59kv\") pod \"klusterlet-addon-workmgr-6cdf68bf66-djbjw\" (UID: \"ffd29531-ff87-48c4-8cd2-8f309ff50e6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.688844 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.688822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l77bx\" (UniqueName: \"kubernetes.io/projected/f35707bf-0576-4f74-9306-2fe4bd43d35e-kube-api-access-l77bx\") pod \"cluster-proxy-proxy-agent-6cf4dfc66-mq5xh\" (UID: \"f35707bf-0576-4f74-9306-2fe4bd43d35e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.689649 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.689622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqst8\" (UniqueName: \"kubernetes.io/projected/30b8ee01-7117-4361-a178-ea00145e6765-kube-api-access-dqst8\") pod \"managed-serviceaccount-addon-agent-56b5665ffd-f7488\" (UID: \"30b8ee01-7117-4361-a178-ea00145e6765\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"
Jul 09 18:05:38.732940 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.732885    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.770678 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.770652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71500497-abe2-44b4-afd0-e956f18e2d01-registry-tls\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.770777 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.770686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71500497-abe2-44b4-afd0-e956f18e2d01-installation-pull-secrets\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.770777 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.770703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71500497-abe2-44b4-afd0-e956f18e2d01-registry-certificates\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.770777 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.770729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71500497-abe2-44b4-afd0-e956f18e2d01-bound-sa-token\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.770975 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.770928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71500497-abe2-44b4-afd0-e956f18e2d01-image-registry-private-configuration\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.771080 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.770984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71500497-abe2-44b4-afd0-e956f18e2d01-ca-trust-extracted\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.771080 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.771043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxmsm\" (UniqueName: \"kubernetes.io/projected/71500497-abe2-44b4-afd0-e956f18e2d01-kube-api-access-dxmsm\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.771080 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.771075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71500497-abe2-44b4-afd0-e956f18e2d01-trusted-ca\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.771297 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.771277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/71500497-abe2-44b4-afd0-e956f18e2d01-ca-trust-extracted\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.771550 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.771530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/71500497-abe2-44b4-afd0-e956f18e2d01-registry-certificates\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.772127 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.772107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/71500497-abe2-44b4-afd0-e956f18e2d01-trusted-ca\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.772997 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.772974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/71500497-abe2-44b4-afd0-e956f18e2d01-installation-pull-secrets\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.773095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.773079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/71500497-abe2-44b4-afd0-e956f18e2d01-image-registry-private-configuration\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.773233 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.773219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/71500497-abe2-44b4-afd0-e956f18e2d01-registry-tls\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.779476 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.779459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxmsm\" (UniqueName: \"kubernetes.io/projected/71500497-abe2-44b4-afd0-e956f18e2d01-kube-api-access-dxmsm\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.780400 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.780382    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71500497-abe2-44b4-afd0-e956f18e2d01-bound-sa-token\") pod \"image-registry-6666f97c54-7dng9\" (UID: \"71500497-abe2-44b4-afd0-e956f18e2d01\") " pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.786345 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.786324    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:38.803706 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.803688    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"
Jul 09 18:05:38.808028 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.808006    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"
Jul 09 18:05:38.834000 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.833858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.899506 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.899474    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:38.938819 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.938743    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"]
Jul 09 18:05:38.941759 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:38.941713    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd29531_ff87_48c4_8cd2_8f309ff50e6c.slice/crio-146b125f544481b11c2a1906ca094f4cb55a329cbd1edc722926069275e8a336 WatchSource:0}: Error finding container 146b125f544481b11c2a1906ca094f4cb55a329cbd1edc722926069275e8a336: Status 404 returned error can't find the container with id 146b125f544481b11c2a1906ca094f4cb55a329cbd1edc722926069275e8a336
Jul 09 18:05:38.956819 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.956788    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.982537 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.982511    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488"]
Jul 09 18:05:38.986126 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:38.986097    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30b8ee01_7117_4361_a178_ea00145e6765.slice/crio-4ec11e12f2e89eaddc0e447062c89d5c2f6682aef21f8b08df351b4736e549d4 WatchSource:0}: Error finding container 4ec11e12f2e89eaddc0e447062c89d5c2f6682aef21f8b08df351b4736e549d4: Status 404 returned error can't find the container with id 4ec11e12f2e89eaddc0e447062c89d5c2f6682aef21f8b08df351b4736e549d4
Jul 09 18:05:38.994563 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:38.994543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh"]
Jul 09 18:05:38.999295 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:38.999269    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf35707bf_0576_4f74_9306_2fe4bd43d35e.slice/crio-480913c2fafa240da61527cf394b66bc7a6dc7bdf8b24c8d3c60686a8c249f9e WatchSource:0}: Error finding container 480913c2fafa240da61527cf394b66bc7a6dc7bdf8b24c8d3c60686a8c249f9e: Status 404 returned error can't find the container with id 480913c2fafa240da61527cf394b66bc7a6dc7bdf8b24c8d3c60686a8c249f9e
Jul 09 18:05:39.053059 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.053034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:39.058816 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.058793    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6666f97c54-7dng9"]
Jul 09 18:05:39.061709 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:39.061687    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71500497_abe2_44b4_afd0_e956f18e2d01.slice/crio-ab0e1df398dd4dc1b933835dce516ae1b0fcbf7ea0c50a764665c9d9ab4c38ef WatchSource:0}: Error finding container ab0e1df398dd4dc1b933835dce516ae1b0fcbf7ea0c50a764665c9d9ab4c38ef: Status 404 returned error can't find the container with id ab0e1df398dd4dc1b933835dce516ae1b0fcbf7ea0c50a764665c9d9ab4c38ef
Jul 09 18:05:39.260126 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.260030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:39.485466 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.485430    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:39.558823 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.558790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488" event={"ID":"30b8ee01-7117-4361-a178-ea00145e6765","Type":"ContainerStarted","Data":"4ec11e12f2e89eaddc0e447062c89d5c2f6682aef21f8b08df351b4736e549d4"}
Jul 09 18:05:39.559956 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.559917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw" event={"ID":"ffd29531-ff87-48c4-8cd2-8f309ff50e6c","Type":"ContainerStarted","Data":"146b125f544481b11c2a1906ca094f4cb55a329cbd1edc722926069275e8a336"}
Jul 09 18:05:39.561880 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.561857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gx8b2" event={"ID":"03a22790-e419-40d7-abdb-6517d9103e1c","Type":"ContainerStarted","Data":"62ecb774f2cbae7e6732109f5f23011e657f832ca361f1841c7f4873beb83e8d"}
Jul 09 18:05:39.563233 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.563209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6666f97c54-7dng9" event={"ID":"71500497-abe2-44b4-afd0-e956f18e2d01","Type":"ContainerStarted","Data":"d0dcca4f2895af335a07e155a7d2e6eefee01567f4d5f95d31b669973fa640c2"}
Jul 09 18:05:39.563348 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.563248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6666f97c54-7dng9" event={"ID":"71500497-abe2-44b4-afd0-e956f18e2d01","Type":"ContainerStarted","Data":"ab0e1df398dd4dc1b933835dce516ae1b0fcbf7ea0c50a764665c9d9ab4c38ef"}
Jul 09 18:05:39.563348 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.563324    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:05:39.564145 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.564127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh" event={"ID":"f35707bf-0576-4f74-9306-2fe4bd43d35e","Type":"ContainerStarted","Data":"480913c2fafa240da61527cf394b66bc7a6dc7bdf8b24c8d3c60686a8c249f9e"}
Jul 09 18:05:39.579805 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.579769    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gx8b2" podStartSLOduration=17.781510345 podStartE2EDuration="25.57975811s" totalImagesPullingTime="7.798247765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:39.578778415 +0000 UTC m=+60.905254124" watchObservedRunningTime="2026-07-09 18:05:39.57975811 +0000 UTC m=+60.906233817"
Jul 09 18:05:39.583877 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.583859    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:39.598743 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.598702    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6666f97c54-7dng9" podStartSLOduration=1.598691322 podStartE2EDuration="1.598691322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:39.59758706 +0000 UTC m=+60.924062764" watchObservedRunningTime="2026-07-09 18:05:39.598691322 +0000 UTC m=+60.925167008"
Jul 09 18:05:39.657128 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:39.657042    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:42.535415 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:42.535378    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9f5n4"
Jul 09 18:05:42.580095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:42.580055    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488" event={"ID":"30b8ee01-7117-4361-a178-ea00145e6765","Type":"ContainerStarted","Data":"eaf2bdddb6c1b668fb5f321baa7c77846d24110a344c40b9500b84f0c377e26f"}
Jul 09 18:05:42.595749 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:42.595703    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56b5665ffd-f7488" podStartSLOduration=1.4492629080000001 podStartE2EDuration="4.595687922s" totalImagesPullingTime="3.146425014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:42.594593063 +0000 UTC m=+63.921068771" watchObservedRunningTime="2026-07-09 18:05:42.595687922 +0000 UTC m=+63.922163629"
Jul 09 18:05:43.022834 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:43.022794    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.124044 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:43.124005    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.200173 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:43.200138    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.499791 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:43.499756    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.877017 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:43.876846    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.932963 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:43.932918    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:44.105746 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:44.105702    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:44.202497 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:44.202463    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:45.128356 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.128321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:45.128731 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.128370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:45.130547 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.130526    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:45.130642 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.130526    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:45.141563 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.141536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d52ad56-9937-4f10-88c3-051fe2f35027-original-pull-secret\") pod \"global-pull-secret-syncer-fxrfv\" (UID: \"5d52ad56-9937-4f10-88c3-051fe2f35027\") " pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:45.141691 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.141669    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641493c8-16d5-46c0-bd0a-8763e005dcdf-metrics-certs\") pod \"network-metrics-daemon-l5m2b\" (UID: \"641493c8-16d5-46c0-bd0a-8763e005dcdf\") " pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:45.228875 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.228842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:45.231031 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.231014    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:45.241575 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.241555    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:45.252511 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.252491    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxgn\" (UniqueName: \"kubernetes.io/projected/1f662f2f-4128-4b2d-be2a-e3053a6f47e5-kube-api-access-wgxgn\") pod \"network-check-target-fp6xc\" (UID: \"1f662f2f-4128-4b2d-be2a-e3053a6f47e5\") " pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:45.323678 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.323655    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pkf52\""
Jul 09 18:05:45.330565 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.330545    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fxrfv"
Jul 09 18:05:45.332672 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.332658    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l5m2b"
Jul 09 18:05:45.338659 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.338641    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6mch4\""
Jul 09 18:05:45.347714 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.347692    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:45.484926 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.484899    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fxrfv"]
Jul 09 18:05:45.489302 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:45.489249    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d52ad56_9937_4f10_88c3_051fe2f35027.slice/crio-376a5748b445a3cf9fc7a313a9f68edf4fdd3314f2671b67498f608adba78877 WatchSource:0}: Error finding container 376a5748b445a3cf9fc7a313a9f68edf4fdd3314f2671b67498f608adba78877: Status 404 returned error can't find the container with id 376a5748b445a3cf9fc7a313a9f68edf4fdd3314f2671b67498f608adba78877
Jul 09 18:05:45.511428 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.511401    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l5m2b"]
Jul 09 18:05:45.513858 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:45.513839    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641493c8_16d5_46c0_bd0a_8763e005dcdf.slice/crio-346c8d47984039b79f842bc87d1892fc6e1a5c4ea73104d62e6813f2d34e2df6 WatchSource:0}: Error finding container 346c8d47984039b79f842bc87d1892fc6e1a5c4ea73104d62e6813f2d34e2df6: Status 404 returned error can't find the container with id 346c8d47984039b79f842bc87d1892fc6e1a5c4ea73104d62e6813f2d34e2df6
Jul 09 18:05:45.522333 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.522311    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fp6xc"]
Jul 09 18:05:45.525865 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:45.525845    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f662f2f_4128_4b2d_be2a_e3053a6f47e5.slice/crio-14e458edf30d4c3f8ad5ae52c4d777519acd67402535fef7906e032532e0ba3d WatchSource:0}: Error finding container 14e458edf30d4c3f8ad5ae52c4d777519acd67402535fef7906e032532e0ba3d: Status 404 returned error can't find the container with id 14e458edf30d4c3f8ad5ae52c4d777519acd67402535fef7906e032532e0ba3d
Jul 09 18:05:45.591983 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.591960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw" event={"ID":"ffd29531-ff87-48c4-8cd2-8f309ff50e6c","Type":"ContainerStarted","Data":"2ec27566fc15919a3d6c376336080f63920002720d20ec78ce8e6b1a24b6336a"}
Jul 09 18:05:45.592175 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.592154    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:45.593184 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.593155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fp6xc" event={"ID":"1f662f2f-4128-4b2d-be2a-e3053a6f47e5","Type":"ContainerStarted","Data":"14e458edf30d4c3f8ad5ae52c4d777519acd67402535fef7906e032532e0ba3d"}
Jul 09 18:05:45.594128 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.594107    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw"
Jul 09 18:05:45.595218 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.595193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh" event={"ID":"f35707bf-0576-4f74-9306-2fe4bd43d35e","Type":"ContainerStarted","Data":"3b0886ef62efb138cbafc82b381c5af15d9695ea41d3d56636eeda36f3526323"}
Jul 09 18:05:45.595349 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.595232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh" event={"ID":"f35707bf-0576-4f74-9306-2fe4bd43d35e","Type":"ContainerStarted","Data":"87986b002a1e0dfabea3fc1d980258e905a2028c10d22c7c602e93546af4cece"}
Jul 09 18:05:45.595349 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.595246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh" event={"ID":"f35707bf-0576-4f74-9306-2fe4bd43d35e","Type":"ContainerStarted","Data":"024b36c742475cb6698bbf5e3c5cc2b2dba69a5417b13f1d7da8bf4f5fd00260"}
Jul 09 18:05:45.598561 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.598537    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l5m2b" event={"ID":"641493c8-16d5-46c0-bd0a-8763e005dcdf","Type":"ContainerStarted","Data":"346c8d47984039b79f842bc87d1892fc6e1a5c4ea73104d62e6813f2d34e2df6"}
Jul 09 18:05:45.599621 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.599600    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fxrfv" event={"ID":"5d52ad56-9937-4f10-88c3-051fe2f35027","Type":"ContainerStarted","Data":"376a5748b445a3cf9fc7a313a9f68edf4fdd3314f2671b67498f608adba78877"}
Jul 09 18:05:45.608309 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.608274    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdf68bf66-djbjw" podStartSLOduration=1.9945930079999998 podStartE2EDuration="7.608261885s" totalImagesPullingTime="5.613668877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:45.607147315 +0000 UTC m=+66.933623021" watchObservedRunningTime="2026-07-09 18:05:45.608261885 +0000 UTC m=+66.934737616"
Jul 09 18:05:45.645194 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:45.645123    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cf4dfc66-mq5xh" podStartSLOduration=2.091221699 podStartE2EDuration="7.645112809s" totalImagesPullingTime="5.55389111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:45.644575987 +0000 UTC m=+66.971051717" watchObservedRunningTime="2026-07-09 18:05:45.645112809 +0000 UTC m=+66.971588514"
Jul 09 18:05:46.225791 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:46.225754    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.301830 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:46.301762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.397285 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:46.397234    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.508110 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:46.507997    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.610743 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:47.610704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l5m2b" event={"ID":"641493c8-16d5-46c0-bd0a-8763e005dcdf","Type":"ContainerStarted","Data":"e2acbf3cdf13d7e6e57c5470d96e6335b6f2216588cf734427cfd3444595b98e"}
Jul 09 18:05:47.611170 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:47.610759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l5m2b" event={"ID":"641493c8-16d5-46c0-bd0a-8763e005dcdf","Type":"ContainerStarted","Data":"c101d85272d5dc19d3008f337aa51dad3c4e871538280fd4207176d28ba7730c"}
Jul 09 18:05:47.632001 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:47.631935    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l5m2b" podStartSLOduration=67.564980103 podStartE2EDuration="1m8.631917937s" totalImagesPullingTime="1.066937834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:47.630257475 +0000 UTC m=+68.956733197" watchObservedRunningTime="2026-07-09 18:05:47.631917937 +0000 UTC m=+68.958393625"
Jul 09 18:05:47.748344 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:47.748297    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.812121 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:47.812093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:48.269604 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:48.269535    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:48.327636 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:48.327601    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:48.471179 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:48.471146    2561 patch_prober.go:28] interesting pod/image-registry-5d8b87f76-79hcx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 18:05:48.471333 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:48.471200    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx" podUID="43cceb62-245b-40a5-b660-c826730cf06a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:05:50.623731 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:50.623635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fxrfv" event={"ID":"5d52ad56-9937-4f10-88c3-051fe2f35027","Type":"ContainerStarted","Data":"27c406815d3a4912a38601aa23ef97eed47e13a3252a6d77d60a166262d54b16"}
Jul 09 18:05:50.625067 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:50.625044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fp6xc" event={"ID":"1f662f2f-4128-4b2d-be2a-e3053a6f47e5","Type":"ContainerStarted","Data":"0b30d3c89eadce17fbc64cc536003d1ea181ce445a7416b12cc5ee43a794a885"}
Jul 09 18:05:50.625133 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:50.625102    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:05:50.653493 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:50.653440    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fp6xc" podStartSLOduration=66.980942069 podStartE2EDuration="1m11.653424604s" totalImagesPullingTime="4.672482535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:50.653168423 +0000 UTC m=+71.979644126" watchObservedRunningTime="2026-07-09 18:05:50.653424604 +0000 UTC m=+71.979900313"
Jul 09 18:05:50.654350 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:50.654315    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fxrfv" podStartSLOduration=66.942156254 podStartE2EDuration="1m11.65430692s" totalImagesPullingTime="4.712150666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:50.638500698 +0000 UTC m=+71.964976411" watchObservedRunningTime="2026-07-09 18:05:50.65430692 +0000 UTC m=+71.980782627"
Jul 09 18:05:51.682210 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:51.682178    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:51.785041 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:51.784999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:53.506246 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:53.506210    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:53.628094 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:53.628066    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:53.998290 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:53.998257    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:55.498602 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:55.498570    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:55.604272 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:55.604219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.283422 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.283392    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.393739 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.393698    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.485549 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.485514    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.601802 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.601691    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.667292 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.667259    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v9lvc"]
Jul 09 18:05:56.670646 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.670620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.675290 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.675267    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:05:56.675473 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.675456    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:05:56.675539 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.675488    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:05:56.675539 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.675511    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:05:56.675784 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.675770    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:05:56.676444 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.676267    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l75qs\""
Jul 09 18:05:56.676444 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.676326    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:05:56.705881 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.705858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.826982 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.826938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nw9z\" (UniqueName: \"kubernetes.io/projected/5a180287-6787-48f3-91d6-4c1ffe89df3a-kube-api-access-6nw9z\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.827131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.827014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a180287-6787-48f3-91d6-4c1ffe89df3a-sys\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.827131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.827039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-tls\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.827131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.827073    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-wtmp\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.827131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.827109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-textfile\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.827131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.827127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-accelerators-collector-config\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.827374 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.827216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a180287-6787-48f3-91d6-4c1ffe89df3a-metrics-client-ca\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.827374 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.827263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.827374 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.827291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a180287-6787-48f3-91d6-4c1ffe89df3a-root\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928336 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928336 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a180287-6787-48f3-91d6-4c1ffe89df3a-root\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928336 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nw9z\" (UniqueName: \"kubernetes.io/projected/5a180287-6787-48f3-91d6-4c1ffe89df3a-kube-api-access-6nw9z\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928583 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a180287-6787-48f3-91d6-4c1ffe89df3a-sys\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928583 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a180287-6787-48f3-91d6-4c1ffe89df3a-sys\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928583 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a180287-6787-48f3-91d6-4c1ffe89df3a-root\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928583 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-tls\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928583 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-wtmp\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928846 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-textfile\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928846 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-accelerators-collector-config\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928846 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a180287-6787-48f3-91d6-4c1ffe89df3a-metrics-client-ca\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928846 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-wtmp\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.928846 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:56.928636    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 18:05:56.928846 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:05:56.928814    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-tls podName:5a180287-6787-48f3-91d6-4c1ffe89df3a nodeName:}" failed. No retries permitted until 2026-07-09 18:05:57.428790102 +0000 UTC m=+78.755265804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-tls") pod "node-exporter-v9lvc" (UID: "5a180287-6787-48f3-91d6-4c1ffe89df3a") : secret "node-exporter-tls" not found
Jul 09 18:05:56.929171 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.928901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-textfile\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.929327 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.929306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a180287-6787-48f3-91d6-4c1ffe89df3a-metrics-client-ca\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.929390 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.929306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-accelerators-collector-config\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.930736 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.930707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:56.932320 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.932298    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.937878 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:56.937857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nw9z\" (UniqueName: \"kubernetes.io/projected/5a180287-6787-48f3-91d6-4c1ffe89df3a-kube-api-access-6nw9z\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:57.034889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.034856    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:57.134247 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.134218    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:57.237444 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.237413    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:57.433296 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.433260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-tls\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:57.435899 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.435862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a180287-6787-48f3-91d6-4c1ffe89df3a-node-exporter-tls\") pod \"node-exporter-v9lvc\" (UID: \"5a180287-6787-48f3-91d6-4c1ffe89df3a\") " pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:57.580908 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.580829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v9lvc"
Jul 09 18:05:57.584874 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.584779    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:57.593371 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:05:57.593344    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a180287_6787_48f3_91d6_4c1ffe89df3a.slice/crio-7ef9c3fa065327ef3e9dcad889733039687afa1a788f73935102f0d1614f3cfc WatchSource:0}: Error finding container 7ef9c3fa065327ef3e9dcad889733039687afa1a788f73935102f0d1614f3cfc: Status 404 returned error can't find the container with id 7ef9c3fa065327ef3e9dcad889733039687afa1a788f73935102f0d1614f3cfc
Jul 09 18:05:57.650723 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.650688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v9lvc" event={"ID":"5a180287-6787-48f3-91d6-4c1ffe89df3a","Type":"ContainerStarted","Data":"7ef9c3fa065327ef3e9dcad889733039687afa1a788f73935102f0d1614f3cfc"}
Jul 09 18:05:57.669367 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.669336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:57.887360 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:57.887285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:58.080798 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:58.080766    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:58.186299 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:58.186267    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:58.469680 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:58.469594    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:05:59.479657 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:59.479621    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:59.580168 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:59.580140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:59.660036 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:59.660007    2561 generic.go:356] "Generic (PLEG): container finished" podID="5a180287-6787-48f3-91d6-4c1ffe89df3a" containerID="4e32b9043ac5c05b6e1cfbb9e91bfbb810093ba717ff451e7b6211faed204247" exitCode=0
Jul 09 18:05:59.660187 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:59.660053    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v9lvc" event={"ID":"5a180287-6787-48f3-91d6-4c1ffe89df3a","Type":"ContainerDied","Data":"4e32b9043ac5c05b6e1cfbb9e91bfbb810093ba717ff451e7b6211faed204247"}
Jul 09 18:05:59.681425 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:59.681394    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:59.895319 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:59.895281    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:59.996742 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:05:59.996652    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:00.574531 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:00.574508    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6666f97c54-7dng9"
Jul 09 18:06:00.665140 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:00.665102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v9lvc" event={"ID":"5a180287-6787-48f3-91d6-4c1ffe89df3a","Type":"ContainerStarted","Data":"4e324eeebb2985f950ba70296d36cd4d6d33c3c8777c03570251c7afd6b1aa2d"}
Jul 09 18:06:00.665298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:00.665151    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v9lvc" event={"ID":"5a180287-6787-48f3-91d6-4c1ffe89df3a","Type":"ContainerStarted","Data":"b27650c1426cd90682bff3c6f51a54b119108579959de6d17d055a75f16d260d"}
Jul 09 18:06:00.685244 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:00.685192    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v9lvc" podStartSLOduration=3.614585893 podStartE2EDuration="4.685173524s" totalImagesPullingTime="1.008859268s" totalInitContainerRuntime="61.728363ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:00.684119827 +0000 UTC m=+82.010595538" watchObservedRunningTime="2026-07-09 18:06:00.685173524 +0000 UTC m=+82.011649234"
Jul 09 18:06:00.701069 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:00.701028    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:00.805694 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:00.805665    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:01.071347 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:01.071199    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:01.133145 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:01.133116    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:01.425964 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:01.425864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:01.530137 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:01.530107    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:01.585877 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:01.585851    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:02.426272 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:02.426240    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:02.501889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:02.501858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:02.922846 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:02.922815    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:03.171070 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.171043    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:03.270634 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.270603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:03.371860 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.371834    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:03.484818 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.484783    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx" podUID="43cceb62-245b-40a5-b660-c826730cf06a" containerName="registry" containerID="cri-o://b3d6e05efdce93c88e949f2cca7e32e74aefa530c80f86258fb63956516a3fd6" gracePeriod=30
Jul 09 18:06:03.676174 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.676146    2561 generic.go:356] "Generic (PLEG): container finished" podID="43cceb62-245b-40a5-b660-c826730cf06a" containerID="b3d6e05efdce93c88e949f2cca7e32e74aefa530c80f86258fb63956516a3fd6" exitCode=0
Jul 09 18:06:03.676294 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.676212    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx" event={"ID":"43cceb62-245b-40a5-b660-c826730cf06a","Type":"ContainerDied","Data":"b3d6e05efdce93c88e949f2cca7e32e74aefa530c80f86258fb63956516a3fd6"}
Jul 09 18:06:03.726916 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.726893    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:06:03.780146 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780118    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-trusted-ca\") pod \"43cceb62-245b-40a5-b660-c826730cf06a\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") "
Jul 09 18:06:03.780254 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780161    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43cceb62-245b-40a5-b660-c826730cf06a-ca-trust-extracted\") pod \"43cceb62-245b-40a5-b660-c826730cf06a\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") "
Jul 09 18:06:03.780254 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc94g\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-kube-api-access-lc94g\") pod \"43cceb62-245b-40a5-b660-c826730cf06a\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") "
Jul 09 18:06:03.780254 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780229    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-installation-pull-secrets\") pod \"43cceb62-245b-40a5-b660-c826730cf06a\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") "
Jul 09 18:06:03.780370 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") pod \"43cceb62-245b-40a5-b660-c826730cf06a\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") "
Jul 09 18:06:03.780370 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780280    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-image-registry-private-configuration\") pod \"43cceb62-245b-40a5-b660-c826730cf06a\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") "
Jul 09 18:06:03.780370 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780302    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-bound-sa-token\") pod \"43cceb62-245b-40a5-b660-c826730cf06a\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") "
Jul 09 18:06:03.780370 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780332    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-registry-certificates\") pod \"43cceb62-245b-40a5-b660-c826730cf06a\" (UID: \"43cceb62-245b-40a5-b660-c826730cf06a\") "
Jul 09 18:06:03.780562 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780463    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "43cceb62-245b-40a5-b660-c826730cf06a" (UID: "43cceb62-245b-40a5-b660-c826730cf06a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:03.780562 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780538    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-trusted-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:06:03.780876 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.780838    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "43cceb62-245b-40a5-b660-c826730cf06a" (UID: "43cceb62-245b-40a5-b660-c826730cf06a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:03.782648 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.782618    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "43cceb62-245b-40a5-b660-c826730cf06a" (UID: "43cceb62-245b-40a5-b660-c826730cf06a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:03.782758 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.782710    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "43cceb62-245b-40a5-b660-c826730cf06a" (UID: "43cceb62-245b-40a5-b660-c826730cf06a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:03.782820 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.782782    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-kube-api-access-lc94g" (OuterVolumeSpecName: "kube-api-access-lc94g") pod "43cceb62-245b-40a5-b660-c826730cf06a" (UID: "43cceb62-245b-40a5-b660-c826730cf06a"). InnerVolumeSpecName "kube-api-access-lc94g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:03.783044 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.783020    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "43cceb62-245b-40a5-b660-c826730cf06a" (UID: "43cceb62-245b-40a5-b660-c826730cf06a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:03.783146 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.783116    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "43cceb62-245b-40a5-b660-c826730cf06a" (UID: "43cceb62-245b-40a5-b660-c826730cf06a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:03.788772 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.788752    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cceb62-245b-40a5-b660-c826730cf06a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "43cceb62-245b-40a5-b660-c826730cf06a" (UID: "43cceb62-245b-40a5-b660-c826730cf06a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:03.884920 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.881616    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc94g\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-kube-api-access-lc94g\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:06:03.884920 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.881672    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-installation-pull-secrets\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:06:03.884920 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.881697    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-registry-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:06:03.884920 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.881728    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43cceb62-245b-40a5-b660-c826730cf06a-image-registry-private-configuration\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:06:03.884920 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.881743    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43cceb62-245b-40a5-b660-c826730cf06a-bound-sa-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:06:03.884920 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.881761    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43cceb62-245b-40a5-b660-c826730cf06a-registry-certificates\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:06:03.884920 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:03.881774    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43cceb62-245b-40a5-b660-c826730cf06a-ca-trust-extracted\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:06:04.680606 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:04.680567    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx" event={"ID":"43cceb62-245b-40a5-b660-c826730cf06a","Type":"ContainerDied","Data":"551836d4f587dfdd71352ef09382e1479affddde17ad363eae2b41c7c2c2c46b"}
Jul 09 18:06:04.681028 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:04.680614    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8b87f76-79hcx"
Jul 09 18:06:04.681028 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:04.680634    2561 scope.go:117] "RemoveContainer" containerID="b3d6e05efdce93c88e949f2cca7e32e74aefa530c80f86258fb63956516a3fd6"
Jul 09 18:06:04.701456 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:04.701433    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d8b87f76-79hcx"]
Jul 09 18:06:04.707143 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:04.707122    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d8b87f76-79hcx"]
Jul 09 18:06:05.312981 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:05.312933    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43cceb62-245b-40a5-b660-c826730cf06a" path="/var/lib/kubelet/pods/43cceb62-245b-40a5-b660-c826730cf06a/volumes"
Jul 09 18:06:06.303249 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:06.303217    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:06.369586 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:06.369561    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:07.228568 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:07.228537    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:07.322128 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:07.322097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:08.523226 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:08.523195    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:08.621113 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:08.621084    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:12.534202 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:12.534170    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:12.635114 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:12.635079    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:12.693985 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:12.693929    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:14.912851 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:14.912811    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:15.013031 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:15.013001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:18.278089 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:18.278056    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:18.331986 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:18.331954    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:21.630127 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:21.630095    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fp6xc"
Jul 09 18:06:23.927769 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:23.927738    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:33.283142 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:33.283105    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:35.028591 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:06:35.028553    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:15.900915 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:15.900879    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:16.122283 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:16.122250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:24.627454 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:24.627420    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:24.682457 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:24.682423    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:24.757789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:24.757759    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:24.817160 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:24.817132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:25.375327 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:25.375301    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:25.443992 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:25.443931    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:35.949692 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:35.949600    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:36.006793 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:36.006756    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:36.994606 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:36.994578    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:47.313382 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:07:47.313351    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:05.412581 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:05.412544    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:05.469344 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:05.469314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:31.126479 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:31.126447    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:31.184000 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:31.183965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:37.683512 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:37.683481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:43.126624 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:43.126585    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:43.212464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:43.212430    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:48.272284 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:48.272253    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:48.365374 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:48.365349    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:48.466486 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:48.466459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:48.566768 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:48.566676    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:48.646971 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:48.646930    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:55.800132 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:08:55.800093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:27.559590 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:09:27.559556    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:38.207876 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:09:38.207840    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:39.185562 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:09:39.185525    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:09:39.189929 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:09:39.189897    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:09:39.193008 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:09:39.192918    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:09:41.330867 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:09:41.330835    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:16.295046 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:16.295012    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:16.394173 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:16.394145    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:21.992225 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:21.992178    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:22.269527 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:22.269455    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:22.440871 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:22.440830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:22.513940 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:22.513908    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:23.159252 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:23.159207    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:23.301594 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:23.301562    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:23.366990 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:23.366939    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:23.468260 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:23.468230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:23.570673 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:23.570646    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:23.720536 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:23.720456    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:23.794683 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:23.794656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:23.996156 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:23.996064    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:24.064423 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:10:24.064389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:27.971544 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:11:27.971491    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:28.042210 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:11:28.042176    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:28.104280 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:11:28.104242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:28.205913 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:11:28.205883    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:28.261968 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:11:28.261879    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:28.847421 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:11:28.847392    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:28.947359 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:11:28.947328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:28.836876 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:28.836841    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:28.896917 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:28.896889    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:28.966398 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:28.966369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:29.031132 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:29.031098    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:29.564995 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:29.564957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:29.619609 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:29.619582    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:37.744886 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:37.744851    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:40.168617 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:40.168582    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:41.796457 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.796429    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:41.843166 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.843134    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-wxbr6"]
Jul 09 18:12:41.843366 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.843355    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cceb62-245b-40a5-b660-c826730cf06a" containerName="registry"
Jul 09 18:12:41.843408 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.843367    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cceb62-245b-40a5-b660-c826730cf06a" containerName="registry"
Jul 09 18:12:41.843440 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.843409    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cceb62-245b-40a5-b660-c826730cf06a" containerName="registry"
Jul 09 18:12:41.846190 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.846165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:41.848603 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.848581    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:12:41.848722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.848604    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-m5rsr\""
Jul 09 18:12:41.849472 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.849452    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:12:41.849574 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.849451    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 09 18:12:41.856299 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.856278    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-wxbr6"]
Jul 09 18:12:41.871937 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.871916    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:41.919241 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.919217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c3aceb9-eb87-4559-9336-c1fbf150f532-cert\") pod \"odh-model-controller-84b54b7f97-wxbr6\" (UID: \"0c3aceb9-eb87-4559-9336-c1fbf150f532\") " pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:41.919336 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.919261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nq6z\" (UniqueName: \"kubernetes.io/projected/0c3aceb9-eb87-4559-9336-c1fbf150f532-kube-api-access-7nq6z\") pod \"odh-model-controller-84b54b7f97-wxbr6\" (UID: \"0c3aceb9-eb87-4559-9336-c1fbf150f532\") " pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:41.930797 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:41.930776    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:42.019726 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.019705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c3aceb9-eb87-4559-9336-c1fbf150f532-cert\") pod \"odh-model-controller-84b54b7f97-wxbr6\" (UID: \"0c3aceb9-eb87-4559-9336-c1fbf150f532\") " pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:42.019830 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.019748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nq6z\" (UniqueName: \"kubernetes.io/projected/0c3aceb9-eb87-4559-9336-c1fbf150f532-kube-api-access-7nq6z\") pod \"odh-model-controller-84b54b7f97-wxbr6\" (UID: \"0c3aceb9-eb87-4559-9336-c1fbf150f532\") " pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:42.019874 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:12:42.019844    2561 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 18:12:42.019920 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:12:42.019906    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3aceb9-eb87-4559-9336-c1fbf150f532-cert podName:0c3aceb9-eb87-4559-9336-c1fbf150f532 nodeName:}" failed. No retries permitted until 2026-07-09 18:12:42.519891964 +0000 UTC m=+483.846367654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0c3aceb9-eb87-4559-9336-c1fbf150f532-cert") pod "odh-model-controller-84b54b7f97-wxbr6" (UID: "0c3aceb9-eb87-4559-9336-c1fbf150f532") : secret "odh-model-controller-webhook-cert" not found
Jul 09 18:12:42.027903 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.027879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nq6z\" (UniqueName: \"kubernetes.io/projected/0c3aceb9-eb87-4559-9336-c1fbf150f532-kube-api-access-7nq6z\") pod \"odh-model-controller-84b54b7f97-wxbr6\" (UID: \"0c3aceb9-eb87-4559-9336-c1fbf150f532\") " pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:42.172119 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.172051    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:42.412653 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.412629    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:42.471314 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.471289    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:42.523701 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.523677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c3aceb9-eb87-4559-9336-c1fbf150f532-cert\") pod \"odh-model-controller-84b54b7f97-wxbr6\" (UID: \"0c3aceb9-eb87-4559-9336-c1fbf150f532\") " pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:42.526000 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.525971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c3aceb9-eb87-4559-9336-c1fbf150f532-cert\") pod \"odh-model-controller-84b54b7f97-wxbr6\" (UID: \"0c3aceb9-eb87-4559-9336-c1fbf150f532\") " pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:42.757309 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.757241    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:42.877259 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.877229    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-wxbr6"]
Jul 09 18:12:42.880418 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:12:42.880383    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3aceb9_eb87_4559_9336_c1fbf150f532.slice/crio-9f5d7e79e429b42cc01f893856423c94a341aef2a6fe370d6d38e0c87cd678fe WatchSource:0}: Error finding container 9f5d7e79e429b42cc01f893856423c94a341aef2a6fe370d6d38e0c87cd678fe: Status 404 returned error can't find the container with id 9f5d7e79e429b42cc01f893856423c94a341aef2a6fe370d6d38e0c87cd678fe
Jul 09 18:12:42.881767 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.881746    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:12:42.948824 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:42.948799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-wxbr6" event={"ID":"0c3aceb9-eb87-4559-9336-c1fbf150f532","Type":"ContainerStarted","Data":"9f5d7e79e429b42cc01f893856423c94a341aef2a6fe370d6d38e0c87cd678fe"}
Jul 09 18:12:46.969055 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:46.969008    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-wxbr6" event={"ID":"0c3aceb9-eb87-4559-9336-c1fbf150f532","Type":"ContainerStarted","Data":"91c71d7b27e02c628e2aab182dfd018e1996a7ae6dfc074c3965981c70c44bce"}
Jul 09 18:12:46.969434 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:46.969104    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:12:46.991387 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:46.991312    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-wxbr6" podStartSLOduration=2.775117468 podStartE2EDuration="5.991294799s" totalImagesPullingTime="3.216177331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:46.990714799 +0000 UTC m=+488.317190511" watchObservedRunningTime="2026-07-09 18:12:46.991294799 +0000 UTC m=+488.317770510"
Jul 09 18:12:57.975080 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:12:57.975050    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-wxbr6"
Jul 09 18:13:11.441051 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:13:11.441017    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:17.775005 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:17.774970    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:17.876059 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:17.876031    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:29.348024 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.347989    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:29.477172 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.477140    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"]
Jul 09 18:14:29.480624 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.480606    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.483053 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.483030    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-e8954-predictor-serving-cert\""
Jul 09 18:14:29.483053 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.483043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:14:29.483406 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.483383    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tfp25\""
Jul 09 18:14:29.483529 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.483408    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:14:29.483529 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.483428    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:14:29.483529 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.483515    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:14:29.483993 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.483973    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-e8954-kube-rbac-proxy-sar-config\""
Jul 09 18:14:29.490816 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.490798    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"]
Jul 09 18:14:29.551926 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.551900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.552034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.551928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddprv\" (UniqueName: \"kubernetes.io/projected/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-api-access-ddprv\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.552034 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.552008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.552118 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.552048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.552118 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.552109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.653409 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.653352    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.653409 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.653381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddprv\" (UniqueName: \"kubernetes.io/projected/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-api-access-ddprv\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.653409 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.653408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.653618 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.653423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.653618 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.653491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.653717 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:14:29.653642    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-e8954-predictor-serving-cert" not found
Jul 09 18:14:29.653764 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:14:29.653726    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-proxy-tls podName:a8f7dd27-1119-4a6a-a1c9-fa3e297d2450 nodeName:}" failed. No retries permitted until 2026-07-09 18:14:30.153704257 +0000 UTC m=+591.480179967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-proxy-tls") pod "isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" (UID: "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450") : secret "isvc-raw-sklearn-batcher-e8954-predictor-serving-cert" not found
Jul 09 18:14:29.653873 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.653853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.654137 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.654120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.654178 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.654122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.664216 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.664196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddprv\" (UniqueName: \"kubernetes.io/projected/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-api-access-ddprv\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:29.708684 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:29.708660    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:14:30.158027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:30.157995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:30.160294 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:30.160263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:30.391140 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:30.391108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:30.525658 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:30.525623    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"]
Jul 09 18:14:30.528629 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:14:30.528603    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f7dd27_1119_4a6a_a1c9_fa3e297d2450.slice/crio-d4ff75cfebaf1f29b11857cd6cfb15fe3ebfb581d3cfd88cebf4eeb40ca17d93 WatchSource:0}: Error finding container d4ff75cfebaf1f29b11857cd6cfb15fe3ebfb581d3cfd88cebf4eeb40ca17d93: Status 404 returned error can't find the container with id d4ff75cfebaf1f29b11857cd6cfb15fe3ebfb581d3cfd88cebf4eeb40ca17d93
Jul 09 18:14:31.307473 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:31.307428    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerStarted","Data":"d4ff75cfebaf1f29b11857cd6cfb15fe3ebfb581d3cfd88cebf4eeb40ca17d93"}
Jul 09 18:14:35.325069 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:35.325026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerStarted","Data":"c021df30abb0b561929df1277ad446db392149b22e426d8308f15555ccbb2289"}
Jul 09 18:14:38.335512 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:38.335421    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerID="c021df30abb0b561929df1277ad446db392149b22e426d8308f15555ccbb2289" exitCode=0
Jul 09 18:14:38.335512 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:38.335494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerDied","Data":"c021df30abb0b561929df1277ad446db392149b22e426d8308f15555ccbb2289"}
Jul 09 18:14:39.225777 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:39.225445    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:14:39.233803 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:39.233702    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:14:52.393279 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:52.393198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerStarted","Data":"debbc09ae917687756896b6d14fdfd8d77b6b3611b6005b8582b59b92c189990"}
Jul 09 18:14:55.407715 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:55.407671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerStarted","Data":"a50f5706bc7cfec3e444ba50c08ac2aa5ad6b7597d1b322edec27ff57fd20d22"}
Jul 09 18:14:57.417488 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:57.417454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerStarted","Data":"45aba7733fc9470ac07f9cfdf8c5d459269eb4de6af1272fd8ecece1cba9c18e"}
Jul 09 18:14:57.417859 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:57.417675    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:57.417859 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:57.417738    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:57.417859 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:57.417770    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:14:57.419147 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:57.419110    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:14:57.419826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:57.419803    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:57.438135 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:57.438086    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podStartSLOduration=2.137056738 podStartE2EDuration="28.438048543s" totalImagesPullingTime="22.819265987s" totalInitContainerRuntime="3.481725818s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:14:29 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:57.437116973 +0000 UTC m=+618.763592682" watchObservedRunningTime="2026-07-09 18:14:57.438048543 +0000 UTC m=+618.764524252"
Jul 09 18:14:58.421805 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:58.421765    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:14:58.422224 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:14:58.422060    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:03.426028 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:03.425992    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:15:03.426617 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:03.426586    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:15:03.426762 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:03.426734    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:13.427239 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:13.427190    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:15:13.427720 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:13.427678    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:15.744815 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:15.744783    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:23.427193 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:23.427149    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:15:23.427664 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:23.427637    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:33.426640 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:33.426591    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:15:33.427086 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:33.427045    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:43.427037 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:43.426987    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:15:43.427455 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:43.427431    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:53.426860 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:53.426808    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:15:53.427427 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:15:53.427287    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:03.426499 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:03.426451    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:16:03.427053 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:03.426778    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:04.309512 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:04.309485    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:16:04.309736 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:04.309720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:16:14.511743 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.511706    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"]
Jul 09 18:16:14.512407 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.512068    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" containerID="cri-o://debbc09ae917687756896b6d14fdfd8d77b6b3611b6005b8582b59b92c189990" gracePeriod=30
Jul 09 18:16:14.512407 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.512254    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy" containerID="cri-o://a50f5706bc7cfec3e444ba50c08ac2aa5ad6b7597d1b322edec27ff57fd20d22" gracePeriod=30
Jul 09 18:16:14.512547 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.512285    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" containerID="cri-o://45aba7733fc9470ac07f9cfdf8c5d459269eb4de6af1272fd8ecece1cba9c18e" gracePeriod=30
Jul 09 18:16:14.545908 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.545878    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:14.636136 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.636106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:14.636320 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.636165    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"]
Jul 09 18:16:14.639260 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.639240    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.652470 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.652446    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-7eb66-predictor-serving-cert\""
Jul 09 18:16:14.652470 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.652466    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-7eb66-kube-rbac-proxy-sar-config\""
Jul 09 18:16:14.664514 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.664478    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"]
Jul 09 18:16:14.678100 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.678070    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerID="a50f5706bc7cfec3e444ba50c08ac2aa5ad6b7597d1b322edec27ff57fd20d22" exitCode=2
Jul 09 18:16:14.678251 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.678142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerDied","Data":"a50f5706bc7cfec3e444ba50c08ac2aa5ad6b7597d1b322edec27ff57fd20d22"}
Jul 09 18:16:14.730507 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.730478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:14.743602 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.743569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sjpm\" (UniqueName: \"kubernetes.io/projected/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-api-access-5sjpm\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.743799 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.743617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.743799 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.743690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.743799 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.743741    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.743799 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.743775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-proxy-tls\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.746134 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.746108    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"]
Jul 09 18:16:14.749324 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.749304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.752648 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.752631    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-7eb66-predictor-serving-cert\""
Jul 09 18:16:14.752898 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.752883    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-7eb66-kube-rbac-proxy-sar-config\""
Jul 09 18:16:14.759818 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.759797    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"]
Jul 09 18:16:14.844395 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sjpm\" (UniqueName: \"kubernetes.io/projected/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-api-access-5sjpm\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.844575 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.844575 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.844575 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.844575 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f090acbf-f787-4a8e-83b0-7a1875c527df-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.844789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.844789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dx4r\" (UniqueName: \"kubernetes.io/projected/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-api-access-4dx4r\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.844789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f090acbf-f787-4a8e-83b0-7a1875c527df-proxy-tls\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.844789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-cabundle-cert\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.844789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.844725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-proxy-tls\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.845058 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.845041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.845509 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.845487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.845605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.845543    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.847598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.847583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-proxy-tls\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.852202 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.852181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sjpm\" (UniqueName: \"kubernetes.io/projected/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-api-access-5sjpm\") pod \"isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.945419 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.945376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.945601 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.945430    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f090acbf-f787-4a8e-83b0-7a1875c527df-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.945601 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.945545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dx4r\" (UniqueName: \"kubernetes.io/projected/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-api-access-4dx4r\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.945601 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.945580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f090acbf-f787-4a8e-83b0-7a1875c527df-proxy-tls\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.945756 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.945609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-cabundle-cert\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.945756 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:16:14.945708    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-serving-cert: secret "isvc-xgboost-graph-raw-7eb66-predictor-serving-cert" not found
Jul 09 18:16:14.945852 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:16:14.945802    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f090acbf-f787-4a8e-83b0-7a1875c527df-proxy-tls podName:f090acbf-f787-4a8e-83b0-7a1875c527df nodeName:}" failed. No retries permitted until 2026-07-09 18:16:15.445773305 +0000 UTC m=+696.772248992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f090acbf-f787-4a8e-83b0-7a1875c527df-proxy-tls") pod "isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" (UID: "f090acbf-f787-4a8e-83b0-7a1875c527df") : secret "isvc-xgboost-graph-raw-7eb66-predictor-serving-cert" not found
Jul 09 18:16:14.945912 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.945845    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f090acbf-f787-4a8e-83b0-7a1875c527df-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.946209 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.946191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-cabundle-cert\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.946268 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.946232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:14.949130 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.949113    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:14.954974 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:14.954928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dx4r\" (UniqueName: \"kubernetes.io/projected/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-api-access-4dx4r\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:15.015278 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:15.015235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:15.077625 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:15.077597    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"]
Jul 09 18:16:15.079845 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:16:15.079810    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a7633f_3c28_46e1_9589_5e11af0e2ae3.slice/crio-ab7fc9cb794d49d8cfd6f2a0de8626573f022d658db533c30a3e54a6c1067a7e WatchSource:0}: Error finding container ab7fc9cb794d49d8cfd6f2a0de8626573f022d658db533c30a3e54a6c1067a7e: Status 404 returned error can't find the container with id ab7fc9cb794d49d8cfd6f2a0de8626573f022d658db533c30a3e54a6c1067a7e
Jul 09 18:16:15.450878 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:15.450820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f090acbf-f787-4a8e-83b0-7a1875c527df-proxy-tls\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:15.453220 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:15.453192    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f090acbf-f787-4a8e-83b0-7a1875c527df-proxy-tls\") pod \"isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:15.660382 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:15.660345    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:15.683557 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:15.683527    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" event={"ID":"c5a7633f-3c28-46e1-9589-5e11af0e2ae3","Type":"ContainerStarted","Data":"9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d"}
Jul 09 18:16:15.683696 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:15.683587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" event={"ID":"c5a7633f-3c28-46e1-9589-5e11af0e2ae3","Type":"ContainerStarted","Data":"ab7fc9cb794d49d8cfd6f2a0de8626573f022d658db533c30a3e54a6c1067a7e"}
Jul 09 18:16:15.785789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:15.785764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"]
Jul 09 18:16:15.787776 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:16:15.787746    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf090acbf_f787_4a8e_83b0_7a1875c527df.slice/crio-3ea8f3f58e47d88fa21d7a4dda70150e3c5e3a1289429bce39bca82c97f6bb14 WatchSource:0}: Error finding container 3ea8f3f58e47d88fa21d7a4dda70150e3c5e3a1289429bce39bca82c97f6bb14: Status 404 returned error can't find the container with id 3ea8f3f58e47d88fa21d7a4dda70150e3c5e3a1289429bce39bca82c97f6bb14
Jul 09 18:16:16.688388 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:16.688347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" event={"ID":"f090acbf-f787-4a8e-83b0-7a1875c527df","Type":"ContainerStarted","Data":"4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389"}
Jul 09 18:16:16.688769 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:16.688423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" event={"ID":"f090acbf-f787-4a8e-83b0-7a1875c527df","Type":"ContainerStarted","Data":"3ea8f3f58e47d88fa21d7a4dda70150e3c5e3a1289429bce39bca82c97f6bb14"}
Jul 09 18:16:18.422162 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:18.422113    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 09 18:16:18.701932 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:18.701905    2561 generic.go:356] "Generic (PLEG): container finished" podID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerID="9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d" exitCode=0
Jul 09 18:16:18.702134 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:18.701978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" event={"ID":"c5a7633f-3c28-46e1-9589-5e11af0e2ae3","Type":"ContainerDied","Data":"9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d"}
Jul 09 18:16:19.708021 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:19.707991    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerID="debbc09ae917687756896b6d14fdfd8d77b6b3611b6005b8582b59b92c189990" exitCode=0
Jul 09 18:16:19.708436 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:19.708065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerDied","Data":"debbc09ae917687756896b6d14fdfd8d77b6b3611b6005b8582b59b92c189990"}
Jul 09 18:16:19.710046 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:19.710020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" event={"ID":"c5a7633f-3c28-46e1-9589-5e11af0e2ae3","Type":"ContainerStarted","Data":"29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139"}
Jul 09 18:16:19.710172 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:19.710069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" event={"ID":"c5a7633f-3c28-46e1-9589-5e11af0e2ae3","Type":"ContainerStarted","Data":"ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c"}
Jul 09 18:16:19.710291 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:19.710277    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:19.711459 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:19.711439    2561 generic.go:356] "Generic (PLEG): container finished" podID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerID="4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389" exitCode=0
Jul 09 18:16:19.711568 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:19.711503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" event={"ID":"f090acbf-f787-4a8e-83b0-7a1875c527df","Type":"ContainerDied","Data":"4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389"}
Jul 09 18:16:19.733313 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:19.733265    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podStartSLOduration=2.219865829 podStartE2EDuration="5.733250396s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.513384567s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:19.732087401 +0000 UTC m=+701.058563110" watchObservedRunningTime="2026-07-09 18:16:19.733250396 +0000 UTC m=+701.059726083"
Jul 09 18:16:20.716311 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:20.716230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:20.717630 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:20.717595    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:16:21.721513 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:21.721162    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:16:23.422447 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:23.422394    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 09 18:16:24.309419 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:24.309376    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:16:24.309796 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:24.309765    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:26.726148 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:26.726121    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:16:26.726633 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:26.726607    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:16:28.422348 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:28.422309    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 09 18:16:28.422823 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:28.422452    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:16:33.422317 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:33.422217    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 09 18:16:34.309663 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:34.309617    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:16:34.310077 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:34.310047    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:36.727071 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:36.727030    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:16:37.786430 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:37.786389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" event={"ID":"f090acbf-f787-4a8e-83b0-7a1875c527df","Type":"ContainerStarted","Data":"762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96"}
Jul 09 18:16:37.786768 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:37.786455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" event={"ID":"f090acbf-f787-4a8e-83b0-7a1875c527df","Type":"ContainerStarted","Data":"9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1"}
Jul 09 18:16:37.786768 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:37.786649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:37.806574 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:37.806496    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podStartSLOduration=2.382147974 podStartE2EDuration="23.806479334s" totalImagesPullingTime="17.903302875s" totalInitContainerRuntime="3.521028485s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:37.805856932 +0000 UTC m=+719.132332641" watchObservedRunningTime="2026-07-09 18:16:37.806479334 +0000 UTC m=+719.132955041"
Jul 09 18:16:38.421871 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:38.421830    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 09 18:16:38.790114 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:38.790080    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:38.790929 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:38.790906    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 09 18:16:39.794025 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:39.793972    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 09 18:16:43.422813 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:43.422773    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Jul 09 18:16:44.309894 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:44.309849    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Jul 09 18:16:44.310125 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:44.310019    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:16:44.310263 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:44.310234    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:44.310360 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:44.310349    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:16:44.798765 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:44.798736    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:16:44.799318 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:44.799293    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 09 18:16:44.815571 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:44.815542    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerID="45aba7733fc9470ac07f9cfdf8c5d459269eb4de6af1272fd8ecece1cba9c18e" exitCode=0
Jul 09 18:16:44.815724 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:44.815625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerDied","Data":"45aba7733fc9470ac07f9cfdf8c5d459269eb4de6af1272fd8ecece1cba9c18e"}
Jul 09 18:16:45.189151 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.189115    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:16:45.307995 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.307886    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddprv\" (UniqueName: \"kubernetes.io/projected/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-api-access-ddprv\") pod \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") "
Jul 09 18:16:45.307995 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.307937    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-rbac-proxy-sar-config\") pod \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") "
Jul 09 18:16:45.307995 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.307969    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-cabundle-cert\") pod \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") "
Jul 09 18:16:45.308290 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.308005    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kserve-provision-location\") pod \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") "
Jul 09 18:16:45.308290 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.308052    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-proxy-tls\") pod \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\" (UID: \"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450\") "
Jul 09 18:16:45.308400 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.308352    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" (UID: "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:45.308455 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.308409    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" (UID: "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:16:45.308455 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.308434    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" (UID: "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:16:45.310389 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.310360    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" (UID: "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:16:45.310518 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.310410    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-api-access-ddprv" (OuterVolumeSpecName: "kube-api-access-ddprv") pod "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" (UID: "a8f7dd27-1119-4a6a-a1c9-fa3e297d2450"). InnerVolumeSpecName "kube-api-access-ddprv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:16:45.409325 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.409287    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddprv\" (UniqueName: \"kubernetes.io/projected/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-api-access-ddprv\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:16:45.409325 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.409317    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:16:45.409534 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.409395    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:16:45.409534 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.409405    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:16:45.409534 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.409414    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:16:45.822166 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.822134    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl" event={"ID":"a8f7dd27-1119-4a6a-a1c9-fa3e297d2450","Type":"ContainerDied","Data":"d4ff75cfebaf1f29b11857cd6cfb15fe3ebfb581d3cfd88cebf4eeb40ca17d93"}
Jul 09 18:16:45.822647 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.822179    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"
Jul 09 18:16:45.822647 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.822214    2561 scope.go:117] "RemoveContainer" containerID="45aba7733fc9470ac07f9cfdf8c5d459269eb4de6af1272fd8ecece1cba9c18e"
Jul 09 18:16:45.829988 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.829971    2561 scope.go:117] "RemoveContainer" containerID="a50f5706bc7cfec3e444ba50c08ac2aa5ad6b7597d1b322edec27ff57fd20d22"
Jul 09 18:16:45.836675 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.836659    2561 scope.go:117] "RemoveContainer" containerID="debbc09ae917687756896b6d14fdfd8d77b6b3611b6005b8582b59b92c189990"
Jul 09 18:16:45.840747 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.840727    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"]
Jul 09 18:16:45.844423 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.844406    2561 scope.go:117] "RemoveContainer" containerID="c021df30abb0b561929df1277ad446db392149b22e426d8308f15555ccbb2289"
Jul 09 18:16:45.845508 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:45.845488    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e8954-predictor-6b8c89c9cc-ql8bl"]
Jul 09 18:16:46.727526 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:46.727490    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:16:47.313255 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:47.313215    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" path="/var/lib/kubelet/pods/a8f7dd27-1119-4a6a-a1c9-fa3e297d2450/volumes"
Jul 09 18:16:54.800137 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:54.800090    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 09 18:16:56.726880 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:16:56.726839    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:17:04.799613 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:04.799573    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 09 18:17:06.727076 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:06.727030    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:17:14.799831 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:14.799785    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 09 18:17:16.726926 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:16.726885    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:17:24.800089 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:24.800051    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 09 18:17:26.726624 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:26.726586    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:17:34.799569 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:34.799523    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 09 18:17:36.728053 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:36.728022    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:17:42.346740 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:42.346709    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:44.800049 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:44.800020    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:17:55.310680 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:55.310630    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:55.410246 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:55.410212    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:56.411386 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:17:56.411352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:04.715637 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.715558    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:04.880520 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.880488    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"]
Jul 09 18:18:04.880771 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.880743    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" containerID="cri-o://ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c" gracePeriod=30
Jul 09 18:18:04.880875 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.880765    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kube-rbac-proxy" containerID="cri-o://29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139" gracePeriod=30
Jul 09 18:18:04.889296 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.889268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:04.935730 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.935691    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"]
Jul 09 18:18:04.936129 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936111    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="storage-initializer"
Jul 09 18:18:04.936190 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936132    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="storage-initializer"
Jul 09 18:18:04.936190 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936159    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container"
Jul 09 18:18:04.936190 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936167    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container"
Jul 09 18:18:04.936190 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936180    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent"
Jul 09 18:18:04.936190 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936191    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent"
Jul 09 18:18:04.936353 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936203    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy"
Jul 09 18:18:04.936353 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936215    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy"
Jul 09 18:18:04.936353 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936278    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kserve-container"
Jul 09 18:18:04.936353 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936291    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="kube-rbac-proxy"
Jul 09 18:18:04.936353 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.936300    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f7dd27-1119-4a6a-a1c9-fa3e297d2450" containerName="agent"
Jul 09 18:18:04.939788 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.939765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:04.942464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.942345    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-4dee8-kube-rbac-proxy-sar-config\""
Jul 09 18:18:04.942464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.942379    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-4dee8-predictor-serving-cert\""
Jul 09 18:18:04.950340 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.950315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"]
Jul 09 18:18:04.974763 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:04.974681    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:05.014782 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.014749    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"]
Jul 09 18:18:05.018073 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.018046    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.020227 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.020207    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-4dee8-kube-rbac-proxy-sar-config\""
Jul 09 18:18:05.020332 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.020207    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-4dee8-predictor-serving-cert\""
Jul 09 18:18:05.024633 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.024609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9x9l\" (UniqueName: \"kubernetes.io/projected/59527165-5357-4056-8dd9-05f5e8a91947-kube-api-access-c9x9l\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.024870 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.024847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59527165-5357-4056-8dd9-05f5e8a91947-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.024937 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.024915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59527165-5357-4056-8dd9-05f5e8a91947-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.025106 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.025078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.025180 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.025153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.029124 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.029099    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"]
Jul 09 18:18:05.050442 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.050392    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:05.117886 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.117850    2561 generic.go:356] "Generic (PLEG): container finished" podID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerID="29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139" exitCode=2
Jul 09 18:18:05.118064 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.117910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" event={"ID":"c5a7633f-3c28-46e1-9589-5e11af0e2ae3","Type":"ContainerDied","Data":"29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139"}
Jul 09 18:18:05.118994 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.118972    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"]
Jul 09 18:18:05.119263 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.119238    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container" containerID="cri-o://9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1" gracePeriod=30
Jul 09 18:18:05.119656 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.119273    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kube-rbac-proxy" containerID="cri-o://762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96" gracePeriod=30
Jul 09 18:18:05.126390 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da73a90-93d4-4ec5-9828-1b62a89a768c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.126494 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.126494 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59527165-5357-4056-8dd9-05f5e8a91947-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.126587 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da73a90-93d4-4ec5-9828-1b62a89a768c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.126587 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:18:05.126549    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-4dee8-predictor-serving-cert" not found
Jul 09 18:18:05.126658 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:18:05.126603    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59527165-5357-4056-8dd9-05f5e8a91947-proxy-tls podName:59527165-5357-4056-8dd9-05f5e8a91947 nodeName:}" failed. No retries permitted until 2026-07-09 18:18:05.626582464 +0000 UTC m=+806.953058157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/59527165-5357-4056-8dd9-05f5e8a91947-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" (UID: "59527165-5357-4056-8dd9-05f5e8a91947") : secret "isvc-sklearn-graph-raw-hpa-4dee8-predictor-serving-cert" not found
Jul 09 18:18:05.126658 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmpxw\" (UniqueName: \"kubernetes.io/projected/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-api-access-zmpxw\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.126741 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.126741 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.126816 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.126816 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9x9l\" (UniqueName: \"kubernetes.io/projected/59527165-5357-4056-8dd9-05f5e8a91947-kube-api-access-c9x9l\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.126917 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.126824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59527165-5357-4056-8dd9-05f5e8a91947-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.127235 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.127213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59527165-5357-4056-8dd9-05f5e8a91947-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.127449 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.127429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.127509 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.127453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.135788 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.135760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9x9l\" (UniqueName: \"kubernetes.io/projected/59527165-5357-4056-8dd9-05f5e8a91947-kube-api-access-c9x9l\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.142083 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.142004    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:05.227598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.227520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da73a90-93d4-4ec5-9828-1b62a89a768c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.227598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.227570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmpxw\" (UniqueName: \"kubernetes.io/projected/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-api-access-zmpxw\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.227598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.227597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.227842 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.227654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da73a90-93d4-4ec5-9828-1b62a89a768c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.227842 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.227682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.228062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.228036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da73a90-93d4-4ec5-9828-1b62a89a768c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.228321 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.228296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.228432 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.228417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.230189 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.230167    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da73a90-93d4-4ec5-9828-1b62a89a768c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.235723 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.235699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmpxw\" (UniqueName: \"kubernetes.io/projected/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-api-access-zmpxw\") pod \"isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.258203 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.258174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:05.328896 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.328868    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:05.461212 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.461183    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"]
Jul 09 18:18:05.463072 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:18:05.463050    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da73a90_93d4_4ec5_9828_1b62a89a768c.slice/crio-4849a900c27fde91248d7309cfcdee2d164c7177206d02b62ac4d2747925235a WatchSource:0}: Error finding container 4849a900c27fde91248d7309cfcdee2d164c7177206d02b62ac4d2747925235a: Status 404 returned error can't find the container with id 4849a900c27fde91248d7309cfcdee2d164c7177206d02b62ac4d2747925235a
Jul 09 18:18:05.489214 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.489161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:05.630491 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.630456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59527165-5357-4056-8dd9-05f5e8a91947-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.632820 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.632788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59527165-5357-4056-8dd9-05f5e8a91947-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:05.853888 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:05.853789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:06.004758 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:06.004721    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"]
Jul 09 18:18:06.008209 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:18:06.008175    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59527165_5357_4056_8dd9_05f5e8a91947.slice/crio-9ea049c3dba380e3013d977f6bf27f24cce950773274e41a0746fad2f6b0254c WatchSource:0}: Error finding container 9ea049c3dba380e3013d977f6bf27f24cce950773274e41a0746fad2f6b0254c: Status 404 returned error can't find the container with id 9ea049c3dba380e3013d977f6bf27f24cce950773274e41a0746fad2f6b0254c
Jul 09 18:18:06.123504 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:06.123422    2561 generic.go:356] "Generic (PLEG): container finished" podID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerID="762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96" exitCode=2
Jul 09 18:18:06.123504 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:06.123491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" event={"ID":"f090acbf-f787-4a8e-83b0-7a1875c527df","Type":"ContainerDied","Data":"762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96"}
Jul 09 18:18:06.124879 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:06.124842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" event={"ID":"59527165-5357-4056-8dd9-05f5e8a91947","Type":"ContainerStarted","Data":"f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf"}
Jul 09 18:18:06.125032 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:06.124897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" event={"ID":"59527165-5357-4056-8dd9-05f5e8a91947","Type":"ContainerStarted","Data":"9ea049c3dba380e3013d977f6bf27f24cce950773274e41a0746fad2f6b0254c"}
Jul 09 18:18:06.126393 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:06.126371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" event={"ID":"7da73a90-93d4-4ec5-9828-1b62a89a768c","Type":"ContainerStarted","Data":"841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0"}
Jul 09 18:18:06.126494 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:06.126429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" event={"ID":"7da73a90-93d4-4ec5-9828-1b62a89a768c","Type":"ContainerStarted","Data":"4849a900c27fde91248d7309cfcdee2d164c7177206d02b62ac4d2747925235a"}
Jul 09 18:18:06.722352 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:06.722310    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 09 18:18:06.726663 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:06.726630    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 09 18:18:09.058231 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.058205    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:18:09.139146 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.139060    2561 generic.go:356] "Generic (PLEG): container finished" podID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerID="9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1" exitCode=0
Jul 09 18:18:09.139146 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.139122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" event={"ID":"f090acbf-f787-4a8e-83b0-7a1875c527df","Type":"ContainerDied","Data":"9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1"}
Jul 09 18:18:09.139348 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.139146    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"
Jul 09 18:18:09.139348 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.139170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z" event={"ID":"f090acbf-f787-4a8e-83b0-7a1875c527df","Type":"ContainerDied","Data":"3ea8f3f58e47d88fa21d7a4dda70150e3c5e3a1289429bce39bca82c97f6bb14"}
Jul 09 18:18:09.139348 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.139197    2561 scope.go:117] "RemoveContainer" containerID="762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96"
Jul 09 18:18:09.147036 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.147015    2561 scope.go:117] "RemoveContainer" containerID="9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1"
Jul 09 18:18:09.154316 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.154301    2561 scope.go:117] "RemoveContainer" containerID="4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389"
Jul 09 18:18:09.159484 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.159459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f090acbf-f787-4a8e-83b0-7a1875c527df-proxy-tls\") pod \"f090acbf-f787-4a8e-83b0-7a1875c527df\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") "
Jul 09 18:18:09.159580 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.159527    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-rbac-proxy-sar-config\") pod \"f090acbf-f787-4a8e-83b0-7a1875c527df\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") "
Jul 09 18:18:09.159580 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.159559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-cabundle-cert\") pod \"f090acbf-f787-4a8e-83b0-7a1875c527df\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") "
Jul 09 18:18:09.159692 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.159587    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dx4r\" (UniqueName: \"kubernetes.io/projected/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-api-access-4dx4r\") pod \"f090acbf-f787-4a8e-83b0-7a1875c527df\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") "
Jul 09 18:18:09.159692 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.159661    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f090acbf-f787-4a8e-83b0-7a1875c527df-kserve-provision-location\") pod \"f090acbf-f787-4a8e-83b0-7a1875c527df\" (UID: \"f090acbf-f787-4a8e-83b0-7a1875c527df\") "
Jul 09 18:18:09.160080 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.160051    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f090acbf-f787-4a8e-83b0-7a1875c527df" (UID: "f090acbf-f787-4a8e-83b0-7a1875c527df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:09.160080 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.160066    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f090acbf-f787-4a8e-83b0-7a1875c527df" (UID: "f090acbf-f787-4a8e-83b0-7a1875c527df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:09.160228 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.160085    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f090acbf-f787-4a8e-83b0-7a1875c527df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f090acbf-f787-4a8e-83b0-7a1875c527df" (UID: "f090acbf-f787-4a8e-83b0-7a1875c527df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:09.161427 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.161404    2561 scope.go:117] "RemoveContainer" containerID="762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96"
Jul 09 18:18:09.161568 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.161547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f090acbf-f787-4a8e-83b0-7a1875c527df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f090acbf-f787-4a8e-83b0-7a1875c527df" (UID: "f090acbf-f787-4a8e-83b0-7a1875c527df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:09.161719 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.161699    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-api-access-4dx4r" (OuterVolumeSpecName: "kube-api-access-4dx4r") pod "f090acbf-f787-4a8e-83b0-7a1875c527df" (UID: "f090acbf-f787-4a8e-83b0-7a1875c527df"). InnerVolumeSpecName "kube-api-access-4dx4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:09.161779 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:18:09.161719    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96\": container with ID starting with 762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96 not found: ID does not exist" containerID="762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96"
Jul 09 18:18:09.161779 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.161748    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96"} err="failed to get container status \"762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96\": rpc error: code = NotFound desc = could not find container \"762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96\": container with ID starting with 762985ccde4468ce720f58744c7f56150d425125aeb5467db8d5130f9b64cf96 not found: ID does not exist"
Jul 09 18:18:09.161779 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.161779    2561 scope.go:117] "RemoveContainer" containerID="9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1"
Jul 09 18:18:09.162066 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:18:09.162019    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1\": container with ID starting with 9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1 not found: ID does not exist" containerID="9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1"
Jul 09 18:18:09.162066 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.162044    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1"} err="failed to get container status \"9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1\": rpc error: code = NotFound desc = could not find container \"9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1\": container with ID starting with 9fb0a704d5144c8874f09c4829dc9e51637fb9af5bed08593158c7b9e33ce4f1 not found: ID does not exist"
Jul 09 18:18:09.162066 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.162060    2561 scope.go:117] "RemoveContainer" containerID="4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389"
Jul 09 18:18:09.162271 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:18:09.162255    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389\": container with ID starting with 4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389 not found: ID does not exist" containerID="4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389"
Jul 09 18:18:09.162331 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.162274    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389"} err="failed to get container status \"4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389\": rpc error: code = NotFound desc = could not find container \"4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389\": container with ID starting with 4faf44d030dcaa8a1b00f38e299960e3cacaff6c91ef1a3e7678f78a6f7ad389 not found: ID does not exist"
Jul 09 18:18:09.260938 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.260904    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f090acbf-f787-4a8e-83b0-7a1875c527df-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.260938 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.260931    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f090acbf-f787-4a8e-83b0-7a1875c527df-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.260938 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.260965    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.261192 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.260974    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f090acbf-f787-4a8e-83b0-7a1875c527df-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.261192 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.260983    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dx4r\" (UniqueName: \"kubernetes.io/projected/f090acbf-f787-4a8e-83b0-7a1875c527df-kube-api-access-4dx4r\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:09.476070 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.476037    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"]
Jul 09 18:18:09.480131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.480095    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7eb66-predictor-7d889fc74f-qgq7z"]
Jul 09 18:18:09.929431 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:09.929404    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:18:10.067262 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.067168    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-cabundle-cert\") pod \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") "
Jul 09 18:18:10.067262 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.067209    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sjpm\" (UniqueName: \"kubernetes.io/projected/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-api-access-5sjpm\") pod \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") "
Jul 09 18:18:10.067262 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.067244    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-proxy-tls\") pod \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") "
Jul 09 18:18:10.067774 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.067301    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-rbac-proxy-sar-config\") pod \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") "
Jul 09 18:18:10.067774 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.067341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kserve-provision-location\") pod \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\" (UID: \"c5a7633f-3c28-46e1-9589-5e11af0e2ae3\") "
Jul 09 18:18:10.067774 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.067595    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5a7633f-3c28-46e1-9589-5e11af0e2ae3" (UID: "c5a7633f-3c28-46e1-9589-5e11af0e2ae3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:10.067774 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.067578    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5a7633f-3c28-46e1-9589-5e11af0e2ae3" (UID: "c5a7633f-3c28-46e1-9589-5e11af0e2ae3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:10.067774 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.067698    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5a7633f-3c28-46e1-9589-5e11af0e2ae3" (UID: "c5a7633f-3c28-46e1-9589-5e11af0e2ae3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:10.069394 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.069372    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-api-access-5sjpm" (OuterVolumeSpecName: "kube-api-access-5sjpm") pod "c5a7633f-3c28-46e1-9589-5e11af0e2ae3" (UID: "c5a7633f-3c28-46e1-9589-5e11af0e2ae3"). InnerVolumeSpecName "kube-api-access-5sjpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:10.069484 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.069448    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5a7633f-3c28-46e1-9589-5e11af0e2ae3" (UID: "c5a7633f-3c28-46e1-9589-5e11af0e2ae3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:10.149272 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.149227    2561 generic.go:356] "Generic (PLEG): container finished" podID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerID="ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c" exitCode=0
Jul 09 18:18:10.149463 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.149305    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"
Jul 09 18:18:10.149463 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.149319    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" event={"ID":"c5a7633f-3c28-46e1-9589-5e11af0e2ae3","Type":"ContainerDied","Data":"ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c"}
Jul 09 18:18:10.149463 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.149421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6" event={"ID":"c5a7633f-3c28-46e1-9589-5e11af0e2ae3","Type":"ContainerDied","Data":"ab7fc9cb794d49d8cfd6f2a0de8626573f022d658db533c30a3e54a6c1067a7e"}
Jul 09 18:18:10.149463 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.149441    2561 scope.go:117] "RemoveContainer" containerID="29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139"
Jul 09 18:18:10.151973 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.151933    2561 generic.go:356] "Generic (PLEG): container finished" podID="59527165-5357-4056-8dd9-05f5e8a91947" containerID="f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf" exitCode=0
Jul 09 18:18:10.152096 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.152012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" event={"ID":"59527165-5357-4056-8dd9-05f5e8a91947","Type":"ContainerDied","Data":"f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf"}
Jul 09 18:18:10.153786 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.153766    2561 generic.go:356] "Generic (PLEG): container finished" podID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerID="841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0" exitCode=0
Jul 09 18:18:10.153877 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.153796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" event={"ID":"7da73a90-93d4-4ec5-9828-1b62a89a768c","Type":"ContainerDied","Data":"841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0"}
Jul 09 18:18:10.158682 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.158601    2561 scope.go:117] "RemoveContainer" containerID="ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c"
Jul 09 18:18:10.168180 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.168155    2561 scope.go:117] "RemoveContainer" containerID="9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d"
Jul 09 18:18:10.168274 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.168211    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:10.168274 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.168227    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sjpm\" (UniqueName: \"kubernetes.io/projected/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-api-access-5sjpm\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:10.168274 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.168236    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:10.168274 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.168245    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:10.168274 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.168257    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a7633f-3c28-46e1-9589-5e11af0e2ae3-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:18:10.177782 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.177768    2561 scope.go:117] "RemoveContainer" containerID="29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139"
Jul 09 18:18:10.178045 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:18:10.178019    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139\": container with ID starting with 29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139 not found: ID does not exist" containerID="29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139"
Jul 09 18:18:10.178134 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.178055    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139"} err="failed to get container status \"29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139\": rpc error: code = NotFound desc = could not find container \"29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139\": container with ID starting with 29562684e2a6e33b703d1c0bcdc5dc2299547fa0ccaa1c4c756c69107c7a4139 not found: ID does not exist"
Jul 09 18:18:10.178134 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.178079    2561 scope.go:117] "RemoveContainer" containerID="ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c"
Jul 09 18:18:10.178347 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:18:10.178329    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c\": container with ID starting with ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c not found: ID does not exist" containerID="ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c"
Jul 09 18:18:10.178388 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.178354    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c"} err="failed to get container status \"ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c\": rpc error: code = NotFound desc = could not find container \"ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c\": container with ID starting with ccba8fe0ff8de6a25ee61bb847c857664c10ec9c1be9e6c152734cb5f69c953c not found: ID does not exist"
Jul 09 18:18:10.178388 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.178372    2561 scope.go:117] "RemoveContainer" containerID="9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d"
Jul 09 18:18:10.178610 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:18:10.178595    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d\": container with ID starting with 9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d not found: ID does not exist" containerID="9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d"
Jul 09 18:18:10.178654 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.178616    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d"} err="failed to get container status \"9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d\": rpc error: code = NotFound desc = could not find container \"9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d\": container with ID starting with 9bd61aed6b1c7af7b42e765e8dff334697948e196742419d67a44583062a154d not found: ID does not exist"
Jul 09 18:18:10.204137 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.204111    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"]
Jul 09 18:18:10.209046 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:10.209027    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7eb66-predictor-7fcffc6765-tzff6"]
Jul 09 18:18:11.161246 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.161197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" event={"ID":"59527165-5357-4056-8dd9-05f5e8a91947","Type":"ContainerStarted","Data":"fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e"}
Jul 09 18:18:11.161713 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.161284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" event={"ID":"59527165-5357-4056-8dd9-05f5e8a91947","Type":"ContainerStarted","Data":"e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403"}
Jul 09 18:18:11.161713 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.161604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:11.163283 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.163264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" event={"ID":"7da73a90-93d4-4ec5-9828-1b62a89a768c","Type":"ContainerStarted","Data":"ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712"}
Jul 09 18:18:11.163353 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.163310    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" event={"ID":"7da73a90-93d4-4ec5-9828-1b62a89a768c","Type":"ContainerStarted","Data":"493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71"}
Jul 09 18:18:11.163495 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.163482    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:11.181217 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.181165    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podStartSLOduration=3.551328968 podStartE2EDuration="7.181149504s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.629820536s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:11.178900895 +0000 UTC m=+812.505376604" watchObservedRunningTime="2026-07-09 18:18:11.181149504 +0000 UTC m=+812.507625213"
Jul 09 18:18:11.197323 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.197278    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podStartSLOduration=3.307454758 podStartE2EDuration="7.197265693s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.889810935s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:11.195826357 +0000 UTC m=+812.522302094" watchObservedRunningTime="2026-07-09 18:18:11.197265693 +0000 UTC m=+812.523741401"
Jul 09 18:18:11.313909 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.313877    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" path="/var/lib/kubelet/pods/c5a7633f-3c28-46e1-9589-5e11af0e2ae3/volumes"
Jul 09 18:18:11.314380 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:11.314367    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" path="/var/lib/kubelet/pods/f090acbf-f787-4a8e-83b0-7a1875c527df/volumes"
Jul 09 18:18:12.167517 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:12.167488    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:12.167895 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:12.167548    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:12.168353 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:12.168330    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:18:12.168471 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:12.168448    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 09 18:18:13.171597 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:13.171555    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:18:13.172087 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:13.171668    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 09 18:18:18.177539 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:18.177499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:18:18.177914 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:18.177610    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:18:18.178165 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:18.178136    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 09 18:18:18.178415 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:18.178391    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:18:28.178744 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:28.178702    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:18:28.179207 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:28.178701    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 09 18:18:35.304623 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:35.304584    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:38.178925 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:38.178879    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:18:38.178925 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:38.178883    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 09 18:18:48.178574 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:48.178535    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 09 18:18:48.178938 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:48.178535    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:18:58.178580 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:58.178535    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:18:58.178580 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:18:58.178535    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 09 18:19:08.178939 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:08.178891    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:19:08.179535 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:08.179075    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:19:18.179170 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:18.179126    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:19:20.309093 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:20.309053    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:19:30.310149 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:30.310105    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:19:39.255183 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:39.255106    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:19:39.256411 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:39.256390    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:19:44.969779 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:44.969750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:45.188958 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:45.188908    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:55.032527 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.032483    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:55.167807 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.167772    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"]
Jul 09 18:19:55.168154 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.168098    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container" containerID="cri-o://e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403" gracePeriod=30
Jul 09 18:19:55.168302 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.168158    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kube-rbac-proxy" containerID="cri-o://fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e" gracePeriod=30
Jul 09 18:19:55.199507 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.199478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:55.245086 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245061    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"]
Jul 09 18:19:55.245381 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245365    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kube-rbac-proxy"
Jul 09 18:19:55.245462 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245385    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kube-rbac-proxy"
Jul 09 18:19:55.245462 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245401    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="storage-initializer"
Jul 09 18:19:55.245462 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245409    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="storage-initializer"
Jul 09 18:19:55.245462 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245432    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container"
Jul 09 18:19:55.245462 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245441    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245467    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kube-rbac-proxy"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245476    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kube-rbac-proxy"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245487    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245495    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245512    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="storage-initializer"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245520    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="storage-initializer"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245588    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kserve-container"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245600    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f090acbf-f787-4a8e-83b0-7a1875c527df" containerName="kube-rbac-proxy"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245612    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kube-rbac-proxy"
Jul 09 18:19:55.245700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.245621    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a7633f-3c28-46e1-9589-5e11af0e2ae3" containerName="kserve-container"
Jul 09 18:19:55.248548 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.248531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.251123 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.251101    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-d927d-predictor-serving-cert\""
Jul 09 18:19:55.251224 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.251125    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-d927d-kube-rbac-proxy-sar-config\""
Jul 09 18:19:55.260453 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.260432    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"]
Jul 09 18:19:55.292743 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.292708    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:55.324773 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.324751    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"]
Jul 09 18:19:55.324973 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.324937    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container" containerID="cri-o://493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71" gracePeriod=30
Jul 09 18:19:55.325032 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.324994    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kube-rbac-proxy" containerID="cri-o://ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712" gracePeriod=30
Jul 09 18:19:55.357872 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.357844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ef69347-296a-4aef-aa32-84f5920b94fd-proxy-tls\") pod \"message-dumper-raw-d927d-predictor-68bccfb784-t7jnl\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") " pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.357987 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.357884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-d927d-predictor-68bccfb784-t7jnl\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") " pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.357987 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.357940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvnz8\" (UniqueName: \"kubernetes.io/projected/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-api-access-gvnz8\") pod \"message-dumper-raw-d927d-predictor-68bccfb784-t7jnl\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") " pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.371492 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.371450    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:55.458482 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.458448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvnz8\" (UniqueName: \"kubernetes.io/projected/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-api-access-gvnz8\") pod \"message-dumper-raw-d927d-predictor-68bccfb784-t7jnl\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") " pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.458593 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.458495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ef69347-296a-4aef-aa32-84f5920b94fd-proxy-tls\") pod \"message-dumper-raw-d927d-predictor-68bccfb784-t7jnl\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") " pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.458593 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.458526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-d927d-predictor-68bccfb784-t7jnl\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") " pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.459329 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.459305    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-d927d-predictor-68bccfb784-t7jnl\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") " pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.460592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.460569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ef69347-296a-4aef-aa32-84f5920b94fd-proxy-tls\") pod \"message-dumper-raw-d927d-predictor-68bccfb784-t7jnl\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") " pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.467228 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.467210    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:55.467489 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.467468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvnz8\" (UniqueName: \"kubernetes.io/projected/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-api-access-gvnz8\") pod \"message-dumper-raw-d927d-predictor-68bccfb784-t7jnl\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") " pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.532335 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.532311    2561 generic.go:356] "Generic (PLEG): container finished" podID="59527165-5357-4056-8dd9-05f5e8a91947" containerID="fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e" exitCode=2
Jul 09 18:19:55.532430 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.532379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" event={"ID":"59527165-5357-4056-8dd9-05f5e8a91947","Type":"ContainerDied","Data":"fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e"}
Jul 09 18:19:55.534297 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.534279    2561 generic.go:356] "Generic (PLEG): container finished" podID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerID="ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712" exitCode=2
Jul 09 18:19:55.534378 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.534346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" event={"ID":"7da73a90-93d4-4ec5-9828-1b62a89a768c","Type":"ContainerDied","Data":"ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712"}
Jul 09 18:19:55.558485 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.558427    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:55.684104 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.684070    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"]
Jul 09 18:19:55.687997 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:19:55.687971    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef69347_296a_4aef_aa32_84f5920b94fd.slice/crio-d48361e89a9dca967ca0741fabb0c051bbc7ae19754b618379306f7574194684 WatchSource:0}: Error finding container d48361e89a9dca967ca0741fabb0c051bbc7ae19754b618379306f7574194684: Status 404 returned error can't find the container with id d48361e89a9dca967ca0741fabb0c051bbc7ae19754b618379306f7574194684
Jul 09 18:19:55.689843 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:55.689828    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:19:56.540125 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:56.540092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl" event={"ID":"6ef69347-296a-4aef-aa32-84f5920b94fd","Type":"ContainerStarted","Data":"d48361e89a9dca967ca0741fabb0c051bbc7ae19754b618379306f7574194684"}
Jul 09 18:19:57.546520 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:57.546482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl" event={"ID":"6ef69347-296a-4aef-aa32-84f5920b94fd","Type":"ContainerStarted","Data":"13dbfec4c8c97dbb476403f435f3b19befb94c3d40be53ede3453913a470a711"}
Jul 09 18:19:57.546974 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:57.546609    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:57.546974 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:57.546665    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:57.546974 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:57.546693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl" event={"ID":"6ef69347-296a-4aef-aa32-84f5920b94fd","Type":"ContainerStarted","Data":"e7651b71186ee956303f549815255dc18fbf9a395089b5049ec664c06a98dbfd"}
Jul 09 18:19:57.548504 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:57.548486    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:19:57.564032 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:57.563988    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl" podStartSLOduration=1.532381524 podStartE2EDuration="2.563974034s" totalImagesPullingTime="1.03159251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:19:57.562921038 +0000 UTC m=+918.889396747" watchObservedRunningTime="2026-07-09 18:19:57.563974034 +0000 UTC m=+918.890449739"
Jul 09 18:19:58.172603 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.172563    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 09 18:19:58.172770 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.172615    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 09 18:19:58.178587 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.178561    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 09 18:19:58.964894 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.964869    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:19:58.983287 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.983266    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da73a90-93d4-4ec5-9828-1b62a89a768c-proxy-tls\") pod \"7da73a90-93d4-4ec5-9828-1b62a89a768c\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") "
Jul 09 18:19:58.983405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.983317    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da73a90-93d4-4ec5-9828-1b62a89a768c-kserve-provision-location\") pod \"7da73a90-93d4-4ec5-9828-1b62a89a768c\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") "
Jul 09 18:19:58.983405 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.983342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmpxw\" (UniqueName: \"kubernetes.io/projected/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-api-access-zmpxw\") pod \"7da73a90-93d4-4ec5-9828-1b62a89a768c\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") "
Jul 09 18:19:58.983521 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.983450    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-cabundle-cert\") pod \"7da73a90-93d4-4ec5-9828-1b62a89a768c\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") "
Jul 09 18:19:58.983521 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.983495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-rbac-proxy-sar-config\") pod \"7da73a90-93d4-4ec5-9828-1b62a89a768c\" (UID: \"7da73a90-93d4-4ec5-9828-1b62a89a768c\") "
Jul 09 18:19:58.983772 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.983743    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da73a90-93d4-4ec5-9828-1b62a89a768c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7da73a90-93d4-4ec5-9828-1b62a89a768c" (UID: "7da73a90-93d4-4ec5-9828-1b62a89a768c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:58.983864 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.983805    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7da73a90-93d4-4ec5-9828-1b62a89a768c" (UID: "7da73a90-93d4-4ec5-9828-1b62a89a768c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:19:58.983864 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.983847    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7da73a90-93d4-4ec5-9828-1b62a89a768c" (UID: "7da73a90-93d4-4ec5-9828-1b62a89a768c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:19:58.985768 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.985656    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7da73a90-93d4-4ec5-9828-1b62a89a768c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7da73a90-93d4-4ec5-9828-1b62a89a768c" (UID: "7da73a90-93d4-4ec5-9828-1b62a89a768c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:19:58.985768 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:58.985738    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-api-access-zmpxw" (OuterVolumeSpecName: "kube-api-access-zmpxw") pod "7da73a90-93d4-4ec5-9828-1b62a89a768c" (UID: "7da73a90-93d4-4ec5-9828-1b62a89a768c"). InnerVolumeSpecName "kube-api-access-zmpxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:19:59.084265 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.084235    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:19:59.084265 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.084261    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7da73a90-93d4-4ec5-9828-1b62a89a768c-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:19:59.084265 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.084271    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da73a90-93d4-4ec5-9828-1b62a89a768c-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:19:59.084470 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.084280    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmpxw\" (UniqueName: \"kubernetes.io/projected/7da73a90-93d4-4ec5-9828-1b62a89a768c-kube-api-access-zmpxw\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:19:59.084470 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.084288    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7da73a90-93d4-4ec5-9828-1b62a89a768c-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:19:59.555285 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.555249    2561 generic.go:356] "Generic (PLEG): container finished" podID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerID="493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71" exitCode=0
Jul 09 18:19:59.555468 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.555308    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"
Jul 09 18:19:59.555468 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.555332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" event={"ID":"7da73a90-93d4-4ec5-9828-1b62a89a768c","Type":"ContainerDied","Data":"493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71"}
Jul 09 18:19:59.555468 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.555403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4" event={"ID":"7da73a90-93d4-4ec5-9828-1b62a89a768c","Type":"ContainerDied","Data":"4849a900c27fde91248d7309cfcdee2d164c7177206d02b62ac4d2747925235a"}
Jul 09 18:19:59.555468 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.555440    2561 scope.go:117] "RemoveContainer" containerID="ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712"
Jul 09 18:19:59.563306 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.563288    2561 scope.go:117] "RemoveContainer" containerID="493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71"
Jul 09 18:19:59.569985 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.569968    2561 scope.go:117] "RemoveContainer" containerID="841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0"
Jul 09 18:19:59.572685 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.572665    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"]
Jul 09 18:19:59.576640 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.576622    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4dee8-predictor-7fc8669d97-kflm4"]
Jul 09 18:19:59.577030 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.577012    2561 scope.go:117] "RemoveContainer" containerID="ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712"
Jul 09 18:19:59.577260 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:19:59.577244    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712\": container with ID starting with ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712 not found: ID does not exist" containerID="ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712"
Jul 09 18:19:59.577320 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.577266    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712"} err="failed to get container status \"ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712\": rpc error: code = NotFound desc = could not find container \"ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712\": container with ID starting with ade828d8a8e9b0a50878cd99019083c52316418a4f1395b482f7e255adfef712 not found: ID does not exist"
Jul 09 18:19:59.577320 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.577290    2561 scope.go:117] "RemoveContainer" containerID="493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71"
Jul 09 18:19:59.577501 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:19:59.577483    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71\": container with ID starting with 493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71 not found: ID does not exist" containerID="493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71"
Jul 09 18:19:59.577562 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.577510    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71"} err="failed to get container status \"493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71\": rpc error: code = NotFound desc = could not find container \"493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71\": container with ID starting with 493cfdd335061e294326cc80e7ee2ef1dd8f8cd3aa75740a11fddf25c26b7d71 not found: ID does not exist"
Jul 09 18:19:59.577562 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.577532    2561 scope.go:117] "RemoveContainer" containerID="841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0"
Jul 09 18:19:59.577755 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:19:59.577739    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0\": container with ID starting with 841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0 not found: ID does not exist" containerID="841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0"
Jul 09 18:19:59.577789 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.577760    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0"} err="failed to get container status \"841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0\": rpc error: code = NotFound desc = could not find container \"841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0\": container with ID starting with 841692127dc1da12efe78a00a695db535a7a384ea50963dc7612bec8087240e0 not found: ID does not exist"
Jul 09 18:19:59.803342 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.803320    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:19:59.889790 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.889715    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-kube-rbac-proxy-sar-config\") pod \"59527165-5357-4056-8dd9-05f5e8a91947\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") "
Jul 09 18:19:59.889790 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.889752    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-cabundle-cert\") pod \"59527165-5357-4056-8dd9-05f5e8a91947\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") "
Jul 09 18:19:59.889790 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.889774    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59527165-5357-4056-8dd9-05f5e8a91947-kserve-provision-location\") pod \"59527165-5357-4056-8dd9-05f5e8a91947\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") "
Jul 09 18:19:59.889999 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.889799    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9x9l\" (UniqueName: \"kubernetes.io/projected/59527165-5357-4056-8dd9-05f5e8a91947-kube-api-access-c9x9l\") pod \"59527165-5357-4056-8dd9-05f5e8a91947\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") "
Jul 09 18:19:59.890084 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.890065    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59527165-5357-4056-8dd9-05f5e8a91947-proxy-tls\") pod \"59527165-5357-4056-8dd9-05f5e8a91947\" (UID: \"59527165-5357-4056-8dd9-05f5e8a91947\") "
Jul 09 18:19:59.890143 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.890074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59527165-5357-4056-8dd9-05f5e8a91947-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59527165-5357-4056-8dd9-05f5e8a91947" (UID: "59527165-5357-4056-8dd9-05f5e8a91947"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:59.890192 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.890171    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "59527165-5357-4056-8dd9-05f5e8a91947" (UID: "59527165-5357-4056-8dd9-05f5e8a91947"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:19:59.890276 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.890250    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59527165-5357-4056-8dd9-05f5e8a91947" (UID: "59527165-5357-4056-8dd9-05f5e8a91947"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:19:59.890379 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.890342    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:19:59.890379 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.890361    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59527165-5357-4056-8dd9-05f5e8a91947-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:19:59.890379 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.890378    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59527165-5357-4056-8dd9-05f5e8a91947-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:19:59.891805 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.891789    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59527165-5357-4056-8dd9-05f5e8a91947-kube-api-access-c9x9l" (OuterVolumeSpecName: "kube-api-access-c9x9l") pod "59527165-5357-4056-8dd9-05f5e8a91947" (UID: "59527165-5357-4056-8dd9-05f5e8a91947"). InnerVolumeSpecName "kube-api-access-c9x9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:19:59.891975 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.891939    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59527165-5357-4056-8dd9-05f5e8a91947-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59527165-5357-4056-8dd9-05f5e8a91947" (UID: "59527165-5357-4056-8dd9-05f5e8a91947"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:19:59.991036 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.991002    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9x9l\" (UniqueName: \"kubernetes.io/projected/59527165-5357-4056-8dd9-05f5e8a91947-kube-api-access-c9x9l\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:19:59.991036 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:19:59.991035    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59527165-5357-4056-8dd9-05f5e8a91947-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:20:00.560445 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.560411    2561 generic.go:356] "Generic (PLEG): container finished" podID="59527165-5357-4056-8dd9-05f5e8a91947" containerID="e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403" exitCode=0
Jul 09 18:20:00.560625 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.560485    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"
Jul 09 18:20:00.560625 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.560486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" event={"ID":"59527165-5357-4056-8dd9-05f5e8a91947","Type":"ContainerDied","Data":"e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403"}
Jul 09 18:20:00.560820 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.560625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr" event={"ID":"59527165-5357-4056-8dd9-05f5e8a91947","Type":"ContainerDied","Data":"9ea049c3dba380e3013d977f6bf27f24cce950773274e41a0746fad2f6b0254c"}
Jul 09 18:20:00.560820 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.560660    2561 scope.go:117] "RemoveContainer" containerID="fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e"
Jul 09 18:20:00.568237 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.568220    2561 scope.go:117] "RemoveContainer" containerID="e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403"
Jul 09 18:20:00.575199 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.575182    2561 scope.go:117] "RemoveContainer" containerID="f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf"
Jul 09 18:20:00.581996 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.581931    2561 scope.go:117] "RemoveContainer" containerID="fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e"
Jul 09 18:20:00.582066 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.582012    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"]
Jul 09 18:20:00.582182 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:20:00.582163    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e\": container with ID starting with fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e not found: ID does not exist" containerID="fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e"
Jul 09 18:20:00.582227 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.582190    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e"} err="failed to get container status \"fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e\": rpc error: code = NotFound desc = could not find container \"fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e\": container with ID starting with fc32e6f440d9de85f59bbc62466f87278ec9d39ecea70b1e385f45b7faa6ca3e not found: ID does not exist"
Jul 09 18:20:00.582227 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.582205    2561 scope.go:117] "RemoveContainer" containerID="e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403"
Jul 09 18:20:00.582431 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:20:00.582417    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403\": container with ID starting with e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403 not found: ID does not exist" containerID="e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403"
Jul 09 18:20:00.582470 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.582433    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403"} err="failed to get container status \"e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403\": rpc error: code = NotFound desc = could not find container \"e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403\": container with ID starting with e11b7bbc731b2155096ad0d57d3c0bfd4292d849e4e1e6fc31db609c4a438403 not found: ID does not exist"
Jul 09 18:20:00.582470 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.582445    2561 scope.go:117] "RemoveContainer" containerID="f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf"
Jul 09 18:20:00.582639 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:20:00.582622    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf\": container with ID starting with f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf not found: ID does not exist" containerID="f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf"
Jul 09 18:20:00.582689 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.582647    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf"} err="failed to get container status \"f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf\": rpc error: code = NotFound desc = could not find container \"f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf\": container with ID starting with f5207c03a2795c74497712fbf5dac136796838585bbbb06edf7270fb8810aecf not found: ID does not exist"
Jul 09 18:20:00.587183 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:00.587165    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4dee8-predictor-75d66c8fc7-slztr"]
Jul 09 18:20:01.312954 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:01.312922    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59527165-5357-4056-8dd9-05f5e8a91947" path="/var/lib/kubelet/pods/59527165-5357-4056-8dd9-05f5e8a91947/volumes"
Jul 09 18:20:01.313429 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:01.313415    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" path="/var/lib/kubelet/pods/7da73a90-93d4-4ec5-9828-1b62a89a768c/volumes"
Jul 09 18:20:04.560042 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:04.560016    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:20:05.257655 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.257623    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"]
Jul 09 18:20:05.257980 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.257923    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container"
Jul 09 18:20:05.257980 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.257938    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container"
Jul 09 18:20:05.257980 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.257964    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container"
Jul 09 18:20:05.257980 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.257983    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.257990    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="storage-initializer"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.257997    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="storage-initializer"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258003    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kube-rbac-proxy"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258008    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kube-rbac-proxy"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258017    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="storage-initializer"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="storage-initializer"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258031    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kube-rbac-proxy"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258036    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kube-rbac-proxy"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258077    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kserve-container"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258083    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kube-rbac-proxy"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258088    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da73a90-93d4-4ec5-9828-1b62a89a768c" containerName="kube-rbac-proxy"
Jul 09 18:20:05.258115 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.258095    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59527165-5357-4056-8dd9-05f5e8a91947" containerName="kserve-container"
Jul 09 18:20:05.260886 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.260869    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.263207 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.263180    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-d927d-predictor-serving-cert\""
Jul 09 18:20:05.263326 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.263182    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:20:05.263326 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.263293    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:20:05.263326 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.263320    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-d927d-kube-rbac-proxy-sar-config\""
Jul 09 18:20:05.271544 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.271522    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"]
Jul 09 18:20:05.330326 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.330306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zhj\" (UniqueName: \"kubernetes.io/projected/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-api-access-m8zhj\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.330433 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.330334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kserve-provision-location\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.330433 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.330354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.330433 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.330370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-cabundle-cert\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.330527 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.330475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e304cb10-26b0-4e39-8268-e0ed9c1eaded-proxy-tls\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.395900 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.395878    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:20:05.431129 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.431104    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zhj\" (UniqueName: \"kubernetes.io/projected/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-api-access-m8zhj\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.431194 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.431134    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kserve-provision-location\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.431194 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.431153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.431194 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.431169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-cabundle-cert\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.431343 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.431212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e304cb10-26b0-4e39-8268-e0ed9c1eaded-proxy-tls\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.431411 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:20:05.431394    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-serving-cert: secret "isvc-logger-raw-d927d-predictor-serving-cert" not found
Jul 09 18:20:05.431497 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:20:05.431485    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e304cb10-26b0-4e39-8268-e0ed9c1eaded-proxy-tls podName:e304cb10-26b0-4e39-8268-e0ed9c1eaded nodeName:}" failed. No retries permitted until 2026-07-09 18:20:05.931466163 +0000 UTC m=+927.257941855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e304cb10-26b0-4e39-8268-e0ed9c1eaded-proxy-tls") pod "isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" (UID: "e304cb10-26b0-4e39-8268-e0ed9c1eaded") : secret "isvc-logger-raw-d927d-predictor-serving-cert" not found
Jul 09 18:20:05.431567 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.431498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kserve-provision-location\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.431773 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.431754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.431906 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.431886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-cabundle-cert\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.439302 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.439283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zhj\" (UniqueName: \"kubernetes.io/projected/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-api-access-m8zhj\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.509879 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.509827    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:20:05.935022 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.934927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e304cb10-26b0-4e39-8268-e0ed9c1eaded-proxy-tls\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:05.937213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:05.937194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e304cb10-26b0-4e39-8268-e0ed9c1eaded-proxy-tls\") pod \"isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:06.171557 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:06.171523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:06.297331 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:06.297305    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"]
Jul 09 18:20:06.583757 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:06.583678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerStarted","Data":"edd745567d638eb2b3def70a38154a7b05e898e1962dd41dc05dcdc844a19223"}
Jul 09 18:20:06.583757 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:06.583741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerStarted","Data":"7893f8e2b63aa72289c76608314ab349f51ed16d8fcfa2eba1dfab1654a1e326"}
Jul 09 18:20:10.604080 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:10.604050    2561 generic.go:356] "Generic (PLEG): container finished" podID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerID="edd745567d638eb2b3def70a38154a7b05e898e1962dd41dc05dcdc844a19223" exitCode=0
Jul 09 18:20:10.604434 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:10.604125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerDied","Data":"edd745567d638eb2b3def70a38154a7b05e898e1962dd41dc05dcdc844a19223"}
Jul 09 18:20:11.609891 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:11.609859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerStarted","Data":"3465c8ff8d9159bc3bae3d65a11f598f308139c984fb3d72a2eab775d644b78e"}
Jul 09 18:20:11.610247 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:11.609922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerStarted","Data":"b101d40a331099b33b7c5a5a139a462399c47d666ffb0ea481ec479c9d735504"}
Jul 09 18:20:11.610247 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:11.609965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerStarted","Data":"0d9b519a2d71a6cc58bba1322c89c572071deb8b36a43f658ff5bf77f16e3a0b"}
Jul 09 18:20:11.610345 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:11.610248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:11.610345 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:11.610286    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:11.610345 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:11.610320    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:11.611814 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:11.611782    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:20:11.612430 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:11.612409    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:11.631758 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:11.631711    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podStartSLOduration=2.868713413 podStartE2EDuration="6.631698389s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.762984976s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:20:11.63008558 +0000 UTC m=+932.956561291" watchObservedRunningTime="2026-07-09 18:20:11.631698389 +0000 UTC m=+932.958174097"
Jul 09 18:20:12.614414 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:12.614377    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:20:12.614823 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:12.614654    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:17.618702 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:17.618671    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:20:17.619265 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:17.619232    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:20:17.619632 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:17.619612    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:27.619683 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:27.619637    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:20:27.620207 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:27.620157    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:37.620024 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:37.619988    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:20:37.620461 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:37.620445    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:47.619855 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:47.619807    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:20:47.620366 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:47.620343    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:57.619199 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:57.619157    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:20:57.619696 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:20:57.619630    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:21:07.619398 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:07.619169    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:21:07.619801 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:07.619574    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:21:17.619936 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:17.619894    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:21:17.620337 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:17.620268    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:21:27.620134 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:27.620098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:21:27.620611 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:27.620594    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:21:40.275474 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.275442    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-d927d-predictor-68bccfb784-t7jnl_6ef69347-296a-4aef-aa32-84f5920b94fd/kserve-container/0.log"
Jul 09 18:21:40.576766 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.576693    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"]
Jul 09 18:21:40.576971 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.576927    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" containerID="cri-o://0d9b519a2d71a6cc58bba1322c89c572071deb8b36a43f658ff5bf77f16e3a0b" gracePeriod=30
Jul 09 18:21:40.577116 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.576993    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" containerID="cri-o://3465c8ff8d9159bc3bae3d65a11f598f308139c984fb3d72a2eab775d644b78e" gracePeriod=30
Jul 09 18:21:40.577116 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.577061    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy" containerID="cri-o://b101d40a331099b33b7c5a5a139a462399c47d666ffb0ea481ec479c9d735504" gracePeriod=30
Jul 09 18:21:40.617775 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.617755    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:40.664025 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.664000    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"]
Jul 09 18:21:40.667434 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.667415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.670086 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.670062    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-94c37-kube-rbac-proxy-sar-config\""
Jul 09 18:21:40.670208 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.670188    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-94c37-predictor-serving-cert\""
Jul 09 18:21:40.678041 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.678019    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"]
Jul 09 18:21:40.709391 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.709367    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"]
Jul 09 18:21:40.709592 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.709573    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl" podUID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerName="kserve-container" containerID="cri-o://e7651b71186ee956303f549815255dc18fbf9a395089b5049ec664c06a98dbfd" gracePeriod=30
Jul 09 18:21:40.709679 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.709653    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl" podUID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerName="kube-rbac-proxy" containerID="cri-o://13dbfec4c8c97dbb476403f435f3b19befb94c3d40be53ede3453913a470a711" gracePeriod=30
Jul 09 18:21:40.715133 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.715108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8zp\" (UniqueName: \"kubernetes.io/projected/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-api-access-lq8zp\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.715230 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.715161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.715230 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.715199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-cabundle-cert\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.715344 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.715252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-proxy-tls\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.715344 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.715277    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.815801 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.815771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-proxy-tls\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.815933 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.815811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.815933 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.815873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8zp\" (UniqueName: \"kubernetes.io/projected/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-api-access-lq8zp\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.815933 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.815910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.816138 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.815989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-cabundle-cert\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.816430 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.816390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.816717 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.816695    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.816782 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.816700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-cabundle-cert\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.818603 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.818583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-proxy-tls\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.824609 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.824588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8zp\" (UniqueName: \"kubernetes.io/projected/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-api-access-lq8zp\") pod \"isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:40.924961 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.924919    2561 generic.go:356] "Generic (PLEG): container finished" podID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerID="b101d40a331099b33b7c5a5a139a462399c47d666ffb0ea481ec479c9d735504" exitCode=2
Jul 09 18:21:40.925112 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.924973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerDied","Data":"b101d40a331099b33b7c5a5a139a462399c47d666ffb0ea481ec479c9d735504"}
Jul 09 18:21:40.926864 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.926844    2561 generic.go:356] "Generic (PLEG): container finished" podID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerID="13dbfec4c8c97dbb476403f435f3b19befb94c3d40be53ede3453913a470a711" exitCode=2
Jul 09 18:21:40.926864 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.926860    2561 generic.go:356] "Generic (PLEG): container finished" podID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerID="e7651b71186ee956303f549815255dc18fbf9a395089b5049ec664c06a98dbfd" exitCode=2
Jul 09 18:21:40.927062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.926883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl" event={"ID":"6ef69347-296a-4aef-aa32-84f5920b94fd","Type":"ContainerDied","Data":"13dbfec4c8c97dbb476403f435f3b19befb94c3d40be53ede3453913a470a711"}
Jul 09 18:21:40.927062 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.926920    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl" event={"ID":"6ef69347-296a-4aef-aa32-84f5920b94fd","Type":"ContainerDied","Data":"e7651b71186ee956303f549815255dc18fbf9a395089b5049ec664c06a98dbfd"}
Jul 09 18:21:40.957343 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.957316    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:21:40.977532 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:40.977510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:41.017414 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.017384    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-rbac-proxy-sar-config\") pod \"6ef69347-296a-4aef-aa32-84f5920b94fd\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") "
Jul 09 18:21:41.017414 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.017420    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvnz8\" (UniqueName: \"kubernetes.io/projected/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-api-access-gvnz8\") pod \"6ef69347-296a-4aef-aa32-84f5920b94fd\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") "
Jul 09 18:21:41.017660 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.017479    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ef69347-296a-4aef-aa32-84f5920b94fd-proxy-tls\") pod \"6ef69347-296a-4aef-aa32-84f5920b94fd\" (UID: \"6ef69347-296a-4aef-aa32-84f5920b94fd\") "
Jul 09 18:21:41.017770 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.017748    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ef69347-296a-4aef-aa32-84f5920b94fd" (UID: "6ef69347-296a-4aef-aa32-84f5920b94fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:21:41.019456 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.019421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-api-access-gvnz8" (OuterVolumeSpecName: "kube-api-access-gvnz8") pod "6ef69347-296a-4aef-aa32-84f5920b94fd" (UID: "6ef69347-296a-4aef-aa32-84f5920b94fd"). InnerVolumeSpecName "kube-api-access-gvnz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:21:41.019456 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.019448    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef69347-296a-4aef-aa32-84f5920b94fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ef69347-296a-4aef-aa32-84f5920b94fd" (UID: "6ef69347-296a-4aef-aa32-84f5920b94fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:21:41.118630 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.118591    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvnz8\" (UniqueName: \"kubernetes.io/projected/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-api-access-gvnz8\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:21:41.118630 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.118618    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ef69347-296a-4aef-aa32-84f5920b94fd-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:21:41.118630 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.118631    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ef69347-296a-4aef-aa32-84f5920b94fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:21:41.123461 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.123435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"]
Jul 09 18:21:41.125302 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:21:41.125237    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0458ea3_132a_4eab_8dcb_d6e4fe28a9ff.slice/crio-94d9c98f36723985cb5427e23882c44032ddf087a5ae07e33088180b93a6996a WatchSource:0}: Error finding container 94d9c98f36723985cb5427e23882c44032ddf087a5ae07e33088180b93a6996a: Status 404 returned error can't find the container with id 94d9c98f36723985cb5427e23882c44032ddf087a5ae07e33088180b93a6996a
Jul 09 18:21:41.932222 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.932144    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"
Jul 09 18:21:41.932222 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.932185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl" event={"ID":"6ef69347-296a-4aef-aa32-84f5920b94fd","Type":"ContainerDied","Data":"d48361e89a9dca967ca0741fabb0c051bbc7ae19754b618379306f7574194684"}
Jul 09 18:21:41.932700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.932265    2561 scope.go:117] "RemoveContainer" containerID="13dbfec4c8c97dbb476403f435f3b19befb94c3d40be53ede3453913a470a711"
Jul 09 18:21:41.933763 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.933730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" event={"ID":"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff","Type":"ContainerStarted","Data":"8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce"}
Jul 09 18:21:41.933852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.933786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" event={"ID":"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff","Type":"ContainerStarted","Data":"94d9c98f36723985cb5427e23882c44032ddf087a5ae07e33088180b93a6996a"}
Jul 09 18:21:41.940473 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.940451    2561 scope.go:117] "RemoveContainer" containerID="e7651b71186ee956303f549815255dc18fbf9a395089b5049ec664c06a98dbfd"
Jul 09 18:21:41.968248 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.968223    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"]
Jul 09 18:21:41.970727 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:41.970706    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d927d-predictor-68bccfb784-t7jnl"]
Jul 09 18:21:42.615364 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:42.615323    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 09 18:21:42.743529 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:42.743498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:43.313901 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:43.313869    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ef69347-296a-4aef-aa32-84f5920b94fd" path="/var/lib/kubelet/pods/6ef69347-296a-4aef-aa32-84f5920b94fd/volumes"
Jul 09 18:21:44.950810 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:44.950785    2561 generic.go:356] "Generic (PLEG): container finished" podID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerID="0d9b519a2d71a6cc58bba1322c89c572071deb8b36a43f658ff5bf77f16e3a0b" exitCode=0
Jul 09 18:21:44.951176 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:44.950861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerDied","Data":"0d9b519a2d71a6cc58bba1322c89c572071deb8b36a43f658ff5bf77f16e3a0b"}
Jul 09 18:21:44.952167 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:44.952145    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerID="8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce" exitCode=0
Jul 09 18:21:44.952262 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:44.952191    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" event={"ID":"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff","Type":"ContainerDied","Data":"8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce"}
Jul 09 18:21:45.958336 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:45.958296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" event={"ID":"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff","Type":"ContainerStarted","Data":"2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad"}
Jul 09 18:21:45.958783 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:45.958371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" event={"ID":"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff","Type":"ContainerStarted","Data":"00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9"}
Jul 09 18:21:45.958783 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:45.958588    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:45.977609 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:45.977567    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podStartSLOduration=2.220725561 podStartE2EDuration="5.977554008s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756828447s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:45.976907036 +0000 UTC m=+1027.303382774" watchObservedRunningTime="2026-07-09 18:21:45.977554008 +0000 UTC m=+1027.304029715"
Jul 09 18:21:46.962142 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:46.962106    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:46.963357 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:46.963335    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:21:47.614831 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:47.614793    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 09 18:21:47.620239 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:47.620216    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:21:47.621434 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:47.621407    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:21:47.965894 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:47.965851    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:21:52.614691 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:52.614655    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 09 18:21:52.615077 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:52.614771    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:21:52.970291 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:52.970265    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:21:52.970861 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:52.970833    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:21:57.614877 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:57.614838    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 09 18:21:57.619187 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:57.619155    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:21:57.621002 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:21:57.620976    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:22:02.615499 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:02.615454    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 09 18:22:02.970763 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:02.970720    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:22:07.615326 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:07.615279    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 09 18:22:07.619702 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:07.619675    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 09 18:22:07.619824 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:07.619805    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:22:07.621237 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:07.621217    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:22:07.621356 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:07.621347    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:22:11.051507 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.051473    2561 generic.go:356] "Generic (PLEG): container finished" podID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerID="3465c8ff8d9159bc3bae3d65a11f598f308139c984fb3d72a2eab775d644b78e" exitCode=0
Jul 09 18:22:11.051852 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.051542    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerDied","Data":"3465c8ff8d9159bc3bae3d65a11f598f308139c984fb3d72a2eab775d644b78e"}
Jul 09 18:22:11.219134 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.219113    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:22:11.345399 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.345326    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e304cb10-26b0-4e39-8268-e0ed9c1eaded-proxy-tls\") pod \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") "
Jul 09 18:22:11.345399 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.345359    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kserve-provision-location\") pod \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") "
Jul 09 18:22:11.345579 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.345404    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8zhj\" (UniqueName: \"kubernetes.io/projected/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-api-access-m8zhj\") pod \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") "
Jul 09 18:22:11.345579 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.345514    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-rbac-proxy-sar-config\") pod \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") "
Jul 09 18:22:11.345660 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.345606    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-cabundle-cert\") pod \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\" (UID: \"e304cb10-26b0-4e39-8268-e0ed9c1eaded\") "
Jul 09 18:22:11.345765 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.345735    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e304cb10-26b0-4e39-8268-e0ed9c1eaded" (UID: "e304cb10-26b0-4e39-8268-e0ed9c1eaded"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:11.345895 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.345844    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e304cb10-26b0-4e39-8268-e0ed9c1eaded" (UID: "e304cb10-26b0-4e39-8268-e0ed9c1eaded"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:22:11.345984 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.345893    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:22:11.346046 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.345983    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e304cb10-26b0-4e39-8268-e0ed9c1eaded" (UID: "e304cb10-26b0-4e39-8268-e0ed9c1eaded"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:22:11.347911 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.347889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e304cb10-26b0-4e39-8268-e0ed9c1eaded-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e304cb10-26b0-4e39-8268-e0ed9c1eaded" (UID: "e304cb10-26b0-4e39-8268-e0ed9c1eaded"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:22:11.347911 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.347892    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-api-access-m8zhj" (OuterVolumeSpecName: "kube-api-access-m8zhj") pod "e304cb10-26b0-4e39-8268-e0ed9c1eaded" (UID: "e304cb10-26b0-4e39-8268-e0ed9c1eaded"). InnerVolumeSpecName "kube-api-access-m8zhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:22:11.446560 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.446527    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:22:11.446560 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.446555    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e304cb10-26b0-4e39-8268-e0ed9c1eaded-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:22:11.446744 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.446567    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zhj\" (UniqueName: \"kubernetes.io/projected/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-api-access-m8zhj\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:22:11.446744 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:11.446578    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e304cb10-26b0-4e39-8268-e0ed9c1eaded-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:22:12.057300 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:12.057271    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"
Jul 09 18:22:12.057679 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:12.057262    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6" event={"ID":"e304cb10-26b0-4e39-8268-e0ed9c1eaded","Type":"ContainerDied","Data":"7893f8e2b63aa72289c76608314ab349f51ed16d8fcfa2eba1dfab1654a1e326"}
Jul 09 18:22:12.057679 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:12.057414    2561 scope.go:117] "RemoveContainer" containerID="3465c8ff8d9159bc3bae3d65a11f598f308139c984fb3d72a2eab775d644b78e"
Jul 09 18:22:12.066615 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:12.066340    2561 scope.go:117] "RemoveContainer" containerID="b101d40a331099b33b7c5a5a139a462399c47d666ffb0ea481ec479c9d735504"
Jul 09 18:22:12.075105 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:12.075088    2561 scope.go:117] "RemoveContainer" containerID="0d9b519a2d71a6cc58bba1322c89c572071deb8b36a43f658ff5bf77f16e3a0b"
Jul 09 18:22:12.082711 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:12.082685    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"]
Jul 09 18:22:12.083083 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:12.083068    2561 scope.go:117] "RemoveContainer" containerID="edd745567d638eb2b3def70a38154a7b05e898e1962dd41dc05dcdc844a19223"
Jul 09 18:22:12.087134 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:12.087113    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d927d-predictor-7c8f5bcc44-wg4p6"]
Jul 09 18:22:12.971669 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:12.971632    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:22:13.313274 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:13.313189    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" path="/var/lib/kubelet/pods/e304cb10-26b0-4e39-8268-e0ed9c1eaded/volumes"
Jul 09 18:22:22.971110 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:22.971072    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:22:32.970996 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:32.970889    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:22:42.971251 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:42.971213    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:22:50.013170 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:50.013112    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:52.971518 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:52.971478    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:22:56.308851 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:22:56.308812    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:23:06.308961 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:23:06.308902    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:23:16.309230 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:23:16.309190    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:23:26.309374 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:23:26.309332    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:23:36.309396 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:23:36.309353    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:23:46.309568 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:23:46.309523    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:23:56.309228 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:23:56.309185    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:24:06.309335 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:06.309236    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:24:16.310110 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:16.310070    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:24:20.779888 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:20.779853    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"]
Jul 09 18:24:20.780339 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:20.780149    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" containerID="cri-o://00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9" gracePeriod=30
Jul 09 18:24:20.780339 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:20.780189    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kube-rbac-proxy" containerID="cri-o://2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad" gracePeriod=30
Jul 09 18:24:20.841527 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:20.841487    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:21.494378 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:21.494342    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerID="2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad" exitCode=2
Jul 09 18:24:21.494554 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:21.494391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" event={"ID":"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff","Type":"ContainerDied","Data":"2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad"}
Jul 09 18:24:22.966894 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:22.966851    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 09 18:24:25.896406 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.896370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:25.955547 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955520    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"]
Jul 09 18:24:25.955779 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955768    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="storage-initializer"
Jul 09 18:24:25.955827 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955782    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="storage-initializer"
Jul 09 18:24:25.955827 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955799    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerName="kube-rbac-proxy"
Jul 09 18:24:25.955827 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955808    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerName="kube-rbac-proxy"
Jul 09 18:24:25.955827 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955815    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent"
Jul 09 18:24:25.955827 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955821    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent"
Jul 09 18:24:25.955827 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955827    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955833    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955841    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerName="kserve-container"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955847    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerName="kserve-container"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955877    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955885    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955932    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerName="kube-rbac-proxy"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955940    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kube-rbac-proxy"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955964    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="kserve-container"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955971    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e304cb10-26b0-4e39-8268-e0ed9c1eaded" containerName="agent"
Jul 09 18:24:25.956027 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.955977    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef69347-296a-4aef-aa32-84f5920b94fd" containerName="kserve-container"
Jul 09 18:24:25.958890 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.958874    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:25.961312 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.961284    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-21c109-predictor-serving-cert\""
Jul 09 18:24:25.961312 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.961306    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-21c109-kube-rbac-proxy-sar-config\""
Jul 09 18:24:25.970404 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:25.970378    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"]
Jul 09 18:24:26.009447 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.009418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpgw\" (UniqueName: \"kubernetes.io/projected/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-api-access-bfpgw\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.009555 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.009456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kserve-provision-location\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.009555 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.009478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-proxy-tls\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.009629 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.009585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-cabundle-cert\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.009629 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.009608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-rbac-proxy-sar-config\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.110446 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.110414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpgw\" (UniqueName: \"kubernetes.io/projected/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-api-access-bfpgw\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.110548 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.110454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kserve-provision-location\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.110548 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.110474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-proxy-tls\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.110548 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.110527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-cabundle-cert\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.110703 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.110551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-rbac-proxy-sar-config\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.110703 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:24:26.110636    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-21c109-predictor-serving-cert: secret "isvc-primary-21c109-predictor-serving-cert" not found
Jul 09 18:24:26.110787 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:24:26.110711    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-proxy-tls podName:a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2 nodeName:}" failed. No retries permitted until 2026-07-09 18:24:26.610691235 +0000 UTC m=+1187.937166925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-proxy-tls") pod "isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" (UID: "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2") : secret "isvc-primary-21c109-predictor-serving-cert" not found
Jul 09 18:24:26.110896 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.110877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kserve-provision-location\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.111226 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.111212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-rbac-proxy-sar-config\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.111277 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.111263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-cabundle-cert\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.119381 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.119363    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpgw\" (UniqueName: \"kubernetes.io/projected/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-api-access-bfpgw\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.308802 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.308761    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 09 18:24:26.318003 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.317968    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:26.615080 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.614987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-proxy-tls\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.617402 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.617381    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-proxy-tls\") pod \"isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") " pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.871348 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.871268    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:26.997171 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:26.997140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"]
Jul 09 18:24:26.999536 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:24:26.999511    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ab2f2e_08ae_4e3e_94c5_6f5ad77812d2.slice/crio-74cfe248b41ea4fd9c68f1ff18fbadb28af3ee21fcfbfbf54286360327926d03 WatchSource:0}: Error finding container 74cfe248b41ea4fd9c68f1ff18fbadb28af3ee21fcfbfbf54286360327926d03: Status 404 returned error can't find the container with id 74cfe248b41ea4fd9c68f1ff18fbadb28af3ee21fcfbfbf54286360327926d03
Jul 09 18:24:27.515235 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:27.515200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" event={"ID":"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2","Type":"ContainerStarted","Data":"445705f629b2d3c83ca982067ca7ea0bf161177e04d5dfaf65bd8087f5d0f3cc"}
Jul 09 18:24:27.515397 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:27.515260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" event={"ID":"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2","Type":"ContainerStarted","Data":"74cfe248b41ea4fd9c68f1ff18fbadb28af3ee21fcfbfbf54286360327926d03"}
Jul 09 18:24:27.966439 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:27.966402    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 09 18:24:30.420209 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.420185    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:24:30.439481 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.439460    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-proxy-tls\") pod \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") "
Jul 09 18:24:30.439577 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.439519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-rbac-proxy-sar-config\") pod \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") "
Jul 09 18:24:30.439577 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.439542    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq8zp\" (UniqueName: \"kubernetes.io/projected/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-api-access-lq8zp\") pod \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") "
Jul 09 18:24:30.439577 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.439565    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-cabundle-cert\") pod \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") "
Jul 09 18:24:30.439738 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.439701    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kserve-provision-location\") pod \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\" (UID: \"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff\") "
Jul 09 18:24:30.439939 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.439906    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" (UID: "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:30.439939 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.439910    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" (UID: "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:30.440220 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.440147    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" (UID: "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:30.442345 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.442317    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" (UID: "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:30.442461 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.442377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-api-access-lq8zp" (OuterVolumeSpecName: "kube-api-access-lq8zp") pod "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" (UID: "d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff"). InnerVolumeSpecName "kube-api-access-lq8zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:30.527313 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.527280    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerID="00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9" exitCode=0
Jul 09 18:24:30.527443 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.527342    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"
Jul 09 18:24:30.527443 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.527356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" event={"ID":"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff","Type":"ContainerDied","Data":"00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9"}
Jul 09 18:24:30.527443 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.527422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl" event={"ID":"d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff","Type":"ContainerDied","Data":"94d9c98f36723985cb5427e23882c44032ddf087a5ae07e33088180b93a6996a"}
Jul 09 18:24:30.527602 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.527454    2561 scope.go:117] "RemoveContainer" containerID="2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad"
Jul 09 18:24:30.535798 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.535776    2561 scope.go:117] "RemoveContainer" containerID="00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9"
Jul 09 18:24:30.540745 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.540728    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:24:30.540829 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.540750    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:24:30.540829 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.540766    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:24:30.540829 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.540778    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq8zp\" (UniqueName: \"kubernetes.io/projected/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-kube-api-access-lq8zp\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:24:30.540829 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.540791    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:24:30.583979 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.583961    2561 scope.go:117] "RemoveContainer" containerID="8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce"
Jul 09 18:24:30.590843 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.590829    2561 scope.go:117] "RemoveContainer" containerID="2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad"
Jul 09 18:24:30.591092 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:24:30.591074    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad\": container with ID starting with 2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad not found: ID does not exist" containerID="2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad"
Jul 09 18:24:30.591142 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.591101    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad"} err="failed to get container status \"2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad\": rpc error: code = NotFound desc = could not find container \"2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad\": container with ID starting with 2be642f4127badfde02a639e2d5100855589e1ae5ca4885cc5fc7f072b2a83ad not found: ID does not exist"
Jul 09 18:24:30.591142 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.591117    2561 scope.go:117] "RemoveContainer" containerID="00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9"
Jul 09 18:24:30.591324 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:24:30.591309    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9\": container with ID starting with 00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9 not found: ID does not exist" containerID="00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9"
Jul 09 18:24:30.591367 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.591330    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9"} err="failed to get container status \"00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9\": rpc error: code = NotFound desc = could not find container \"00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9\": container with ID starting with 00c5ad7d990a062324ac441eba1b5dd55f0fb11e304e079756c1ae00d8efb7d9 not found: ID does not exist"
Jul 09 18:24:30.591367 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.591348    2561 scope.go:117] "RemoveContainer" containerID="8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce"
Jul 09 18:24:30.591579 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:24:30.591562    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce\": container with ID starting with 8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce not found: ID does not exist" containerID="8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce"
Jul 09 18:24:30.591629 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.591587    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce"} err="failed to get container status \"8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce\": rpc error: code = NotFound desc = could not find container \"8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce\": container with ID starting with 8288c250275646af5a95904214e254200f5dc780c556a433bff66cd50ebee7ce not found: ID does not exist"
Jul 09 18:24:30.605874 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.605851    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"]
Jul 09 18:24:30.618755 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:30.618737    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-94c37-predictor-566cd5685f-hgvxl"]
Jul 09 18:24:31.313678 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:31.313643    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" path="/var/lib/kubelet/pods/d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff/volumes"
Jul 09 18:24:31.533487 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:31.533420    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerID="445705f629b2d3c83ca982067ca7ea0bf161177e04d5dfaf65bd8087f5d0f3cc" exitCode=0
Jul 09 18:24:31.533487 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:31.533474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" event={"ID":"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2","Type":"ContainerDied","Data":"445705f629b2d3c83ca982067ca7ea0bf161177e04d5dfaf65bd8087f5d0f3cc"}
Jul 09 18:24:32.539807 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:32.539771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" event={"ID":"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2","Type":"ContainerStarted","Data":"561e74e5bcf5789fcd6d381e202c652e83bdde071d1c6a9549e1c849d7b57661"}
Jul 09 18:24:32.540288 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:32.539830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" event={"ID":"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2","Type":"ContainerStarted","Data":"c5bcce49bb6e15e50cc61a64f1b10febf76b88c55b56aba4b18cb8a25d56823f"}
Jul 09 18:24:32.540288 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:32.540073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:32.560268 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:32.560219    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podStartSLOduration=4.083995054 podStartE2EDuration="7.560205644s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.47621059s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:32.558973147 +0000 UTC m=+1193.885448849" watchObservedRunningTime="2026-07-09 18:24:32.560205644 +0000 UTC m=+1193.886681352"
Jul 09 18:24:33.543147 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:33.543111    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:33.544368 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:33.544338    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 09 18:24:34.547982 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:34.547907    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 09 18:24:39.276209 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:39.276182    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:24:39.276209 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:39.276196    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:24:39.551746 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:39.551673    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:24:39.552138 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:39.552108    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 09 18:24:49.552625 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:49.552589    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 09 18:24:59.552191 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:24:59.552154    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 09 18:25:09.552187 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:09.552146    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 09 18:25:19.552119 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:19.552074    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 09 18:25:29.552761 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:29.552719    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 09 18:25:39.552170 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:39.552096    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 09 18:25:49.552673 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:49.552646    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:25:56.099719 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.099682    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:56.169775 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.169744    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"]
Jul 09 18:25:56.170025 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.170011    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container"
Jul 09 18:25:56.170071 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.170027    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container"
Jul 09 18:25:56.170071 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.170041    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="storage-initializer"
Jul 09 18:25:56.170071 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.170046    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="storage-initializer"
Jul 09 18:25:56.170071 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.170054    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kube-rbac-proxy"
Jul 09 18:25:56.170071 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.170060    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kube-rbac-proxy"
Jul 09 18:25:56.170211 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.170106    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kserve-container"
Jul 09 18:25:56.170211 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.170113    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0458ea3-132a-4eab-8dcb-d6e4fe28a9ff" containerName="kube-rbac-proxy"
Jul 09 18:25:56.172922 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.172906    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.175923 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.175903    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-21c109-dockercfg-hrg6v\""
Jul 09 18:25:56.176051 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.175975    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-21c109-predictor-serving-cert\""
Jul 09 18:25:56.176183 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.176168    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-21c109\""
Jul 09 18:25:56.176183 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.176177    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-21c109-kube-rbac-proxy-sar-config\""
Jul 09 18:25:56.184616 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.184598    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"]
Jul 09 18:25:56.248844 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.248816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kserve-provision-location\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.248976 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.248849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-cabundle-cert\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.248976 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.248882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.248976 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.248932    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50aa2df8-e0a7-4631-b285-cb0ad984ef24-proxy-tls\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.249140 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.249021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwc8\" (UniqueName: \"kubernetes.io/projected/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-api-access-vkwc8\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.249379 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.249361    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:56.350037 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.349967    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwc8\" (UniqueName: \"kubernetes.io/projected/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-api-access-vkwc8\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.350037 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.350006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kserve-provision-location\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.350037 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.350024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-cabundle-cert\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.350251 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.350061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.350251 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.350089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50aa2df8-e0a7-4631-b285-cb0ad984ef24-proxy-tls\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.350456 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.350435    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kserve-provision-location\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.350860 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.350837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-cabundle-cert\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.350898 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.350837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.352362 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.352347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50aa2df8-e0a7-4631-b285-cb0ad984ef24-proxy-tls\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.366310 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.366287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwc8\" (UniqueName: \"kubernetes.io/projected/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-api-access-vkwc8\") pod \"isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") " pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.484593 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.484557    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:25:56.613873 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.613799    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"]
Jul 09 18:25:56.616802 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:25:56.616772    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50aa2df8_e0a7_4631_b285_cb0ad984ef24.slice/crio-c0b0be97d26d441f244f5b5623ed45b8ee32611f160fbe0906061930e8a7281f WatchSource:0}: Error finding container c0b0be97d26d441f244f5b5623ed45b8ee32611f160fbe0906061930e8a7281f: Status 404 returned error can't find the container with id c0b0be97d26d441f244f5b5623ed45b8ee32611f160fbe0906061930e8a7281f
Jul 09 18:25:56.836207 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.836162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w" event={"ID":"50aa2df8-e0a7-4631-b285-cb0ad984ef24","Type":"ContainerStarted","Data":"0aeabc5e216ad08196a059885b9cb5ad03f5edec6b6b7aea6e0b0cf62f79782c"}
Jul 09 18:25:56.836406 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:25:56.836233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w" event={"ID":"50aa2df8-e0a7-4631-b285-cb0ad984ef24","Type":"ContainerStarted","Data":"c0b0be97d26d441f244f5b5623ed45b8ee32611f160fbe0906061930e8a7281f"}
Jul 09 18:26:02.859452 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:02.859425    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_50aa2df8-e0a7-4631-b285-cb0ad984ef24/storage-initializer/0.log"
Jul 09 18:26:02.859862 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:02.859460    2561 generic.go:356] "Generic (PLEG): container finished" podID="50aa2df8-e0a7-4631-b285-cb0ad984ef24" containerID="0aeabc5e216ad08196a059885b9cb5ad03f5edec6b6b7aea6e0b0cf62f79782c" exitCode=1
Jul 09 18:26:02.859862 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:02.859529    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w" event={"ID":"50aa2df8-e0a7-4631-b285-cb0ad984ef24","Type":"ContainerDied","Data":"0aeabc5e216ad08196a059885b9cb5ad03f5edec6b6b7aea6e0b0cf62f79782c"}
Jul 09 18:26:03.864761 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:03.864734    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_50aa2df8-e0a7-4631-b285-cb0ad984ef24/storage-initializer/0.log"
Jul 09 18:26:03.865150 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:03.864782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w" event={"ID":"50aa2df8-e0a7-4631-b285-cb0ad984ef24","Type":"ContainerStarted","Data":"0c3c44832d60443b8b762848c53a2e0840bd69c528aac52ead49ca32134b5b9d"}
Jul 09 18:26:05.872161 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:05.872135    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_50aa2df8-e0a7-4631-b285-cb0ad984ef24/storage-initializer/1.log"
Jul 09 18:26:05.872543 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:05.872475    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_50aa2df8-e0a7-4631-b285-cb0ad984ef24/storage-initializer/0.log"
Jul 09 18:26:05.872543 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:05.872503    2561 generic.go:356] "Generic (PLEG): container finished" podID="50aa2df8-e0a7-4631-b285-cb0ad984ef24" containerID="0c3c44832d60443b8b762848c53a2e0840bd69c528aac52ead49ca32134b5b9d" exitCode=1
Jul 09 18:26:05.872642 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:05.872564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w" event={"ID":"50aa2df8-e0a7-4631-b285-cb0ad984ef24","Type":"ContainerDied","Data":"0c3c44832d60443b8b762848c53a2e0840bd69c528aac52ead49ca32134b5b9d"}
Jul 09 18:26:05.872642 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:05.872618    2561 scope.go:117] "RemoveContainer" containerID="0aeabc5e216ad08196a059885b9cb5ad03f5edec6b6b7aea6e0b0cf62f79782c"
Jul 09 18:26:05.873018 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:05.872994    2561 scope.go:117] "RemoveContainer" containerID="0aeabc5e216ad08196a059885b9cb5ad03f5edec6b6b7aea6e0b0cf62f79782c"
Jul 09 18:26:05.882597 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:26:05.882577    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_kserve-ci-e2e-test(50aa2df8-e0a7-4631-b285-cb0ad984ef24)\"" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w" podUID="50aa2df8-e0a7-4631-b285-cb0ad984ef24"
Jul 09 18:26:05.882770 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:26:05.882741    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_kserve-ci-e2e-test_50aa2df8-e0a7-4631-b285-cb0ad984ef24_0 in pod sandbox c0b0be97d26d441f244f5b5623ed45b8ee32611f160fbe0906061930e8a7281f from index: no such id: '0aeabc5e216ad08196a059885b9cb5ad03f5edec6b6b7aea6e0b0cf62f79782c'" containerID="0aeabc5e216ad08196a059885b9cb5ad03f5edec6b6b7aea6e0b0cf62f79782c"
Jul 09 18:26:05.882865 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:05.882775    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aeabc5e216ad08196a059885b9cb5ad03f5edec6b6b7aea6e0b0cf62f79782c"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_kserve-ci-e2e-test_50aa2df8-e0a7-4631-b285-cb0ad984ef24_0 in pod sandbox c0b0be97d26d441f244f5b5623ed45b8ee32611f160fbe0906061930e8a7281f from index: no such id: '0aeabc5e216ad08196a059885b9cb5ad03f5edec6b6b7aea6e0b0cf62f79782c'"
Jul 09 18:26:06.878095 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:06.878064    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_50aa2df8-e0a7-4631-b285-cb0ad984ef24/storage-initializer/1.log"
Jul 09 18:26:14.199864 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.199833    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"]
Jul 09 18:26:14.225441 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.225410    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:14.261499 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.261466    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"]
Jul 09 18:26:14.261769 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.261714    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container" containerID="cri-o://c5bcce49bb6e15e50cc61a64f1b10febf76b88c55b56aba4b18cb8a25d56823f" gracePeriod=30
Jul 09 18:26:14.261769 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.261733    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kube-rbac-proxy" containerID="cri-o://561e74e5bcf5789fcd6d381e202c652e83bdde071d1c6a9549e1c849d7b57661" gracePeriod=30
Jul 09 18:26:14.304172 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.304149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:14.380743 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.380723    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_50aa2df8-e0a7-4631-b285-cb0ad984ef24/storage-initializer/1.log"
Jul 09 18:26:14.382772 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.382758    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:26:14.480135 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.480051    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkwc8\" (UniqueName: \"kubernetes.io/projected/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-api-access-vkwc8\") pod \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") "
Jul 09 18:26:14.480135 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.480103    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-rbac-proxy-sar-config\") pod \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") "
Jul 09 18:26:14.480135 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.480127    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50aa2df8-e0a7-4631-b285-cb0ad984ef24-proxy-tls\") pod \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") "
Jul 09 18:26:14.480404 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.480145    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-cabundle-cert\") pod \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") "
Jul 09 18:26:14.480404 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.480182    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kserve-provision-location\") pod \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\" (UID: \"50aa2df8-e0a7-4631-b285-cb0ad984ef24\") "
Jul 09 18:26:14.480522 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.480498    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50aa2df8-e0a7-4631-b285-cb0ad984ef24" (UID: "50aa2df8-e0a7-4631-b285-cb0ad984ef24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:14.480575 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.480546    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50aa2df8-e0a7-4631-b285-cb0ad984ef24" (UID: "50aa2df8-e0a7-4631-b285-cb0ad984ef24"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:14.480575 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.480553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50aa2df8-e0a7-4631-b285-cb0ad984ef24" (UID: "50aa2df8-e0a7-4631-b285-cb0ad984ef24"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:14.482269 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.482236    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-api-access-vkwc8" (OuterVolumeSpecName: "kube-api-access-vkwc8") pod "50aa2df8-e0a7-4631-b285-cb0ad984ef24" (UID: "50aa2df8-e0a7-4631-b285-cb0ad984ef24"). InnerVolumeSpecName "kube-api-access-vkwc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:14.482378 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.482274    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50aa2df8-e0a7-4631-b285-cb0ad984ef24-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50aa2df8-e0a7-4631-b285-cb0ad984ef24" (UID: "50aa2df8-e0a7-4631-b285-cb0ad984ef24"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:14.548851 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.548804    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 09 18:26:14.581642 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.581608    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkwc8\" (UniqueName: \"kubernetes.io/projected/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-api-access-vkwc8\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:14.581642 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.581638    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:14.581642 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.581648    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50aa2df8-e0a7-4631-b285-cb0ad984ef24-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:14.581860 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.581657    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50aa2df8-e0a7-4631-b285-cb0ad984ef24-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:14.581860 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.581665    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50aa2df8-e0a7-4631-b285-cb0ad984ef24-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:14.906886 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.906808    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerID="561e74e5bcf5789fcd6d381e202c652e83bdde071d1c6a9549e1c849d7b57661" exitCode=2
Jul 09 18:26:14.907045 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.906880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" event={"ID":"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2","Type":"ContainerDied","Data":"561e74e5bcf5789fcd6d381e202c652e83bdde071d1c6a9549e1c849d7b57661"}
Jul 09 18:26:14.908039 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.908024    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w_50aa2df8-e0a7-4631-b285-cb0ad984ef24/storage-initializer/1.log"
Jul 09 18:26:14.908149 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.908110    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"
Jul 09 18:26:14.908195 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.908106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w" event={"ID":"50aa2df8-e0a7-4631-b285-cb0ad984ef24","Type":"ContainerDied","Data":"c0b0be97d26d441f244f5b5623ed45b8ee32611f160fbe0906061930e8a7281f"}
Jul 09 18:26:14.908246 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.908235    2561 scope.go:117] "RemoveContainer" containerID="0c3c44832d60443b8b762848c53a2e0840bd69c528aac52ead49ca32134b5b9d"
Jul 09 18:26:14.945704 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.945677    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"]
Jul 09 18:26:14.949541 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:14.949513    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-21c109-predictor-8b5c6bc6c-xpm9w"]
Jul 09 18:26:15.313661 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:15.313630    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50aa2df8-e0a7-4631-b285-cb0ad984ef24" path="/var/lib/kubelet/pods/50aa2df8-e0a7-4631-b285-cb0ad984ef24/volumes"
Jul 09 18:26:18.927313 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:18.925022    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerID="c5bcce49bb6e15e50cc61a64f1b10febf76b88c55b56aba4b18cb8a25d56823f" exitCode=0
Jul 09 18:26:18.927313 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:18.925074    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" event={"ID":"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2","Type":"ContainerDied","Data":"c5bcce49bb6e15e50cc61a64f1b10febf76b88c55b56aba4b18cb8a25d56823f"}
Jul 09 18:26:19.013785 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.013760    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:26:19.016769 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.016747    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-rbac-proxy-sar-config\") pod \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") "
Jul 09 18:26:19.016889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.016780    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfpgw\" (UniqueName: \"kubernetes.io/projected/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-api-access-bfpgw\") pod \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") "
Jul 09 18:26:19.016889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.016818    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-cabundle-cert\") pod \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") "
Jul 09 18:26:19.016889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.016850    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kserve-provision-location\") pod \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") "
Jul 09 18:26:19.016889 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.016866    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-proxy-tls\") pod \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\" (UID: \"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2\") "
Jul 09 18:26:19.017119 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.017084    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" (UID: "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:19.017220 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.017201    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" (UID: "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:19.017416 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.017224    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" (UID: "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:19.018899 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.018877    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-api-access-bfpgw" (OuterVolumeSpecName: "kube-api-access-bfpgw") pod "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" (UID: "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2"). InnerVolumeSpecName "kube-api-access-bfpgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:19.018899 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.018889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" (UID: "a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:19.117911 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.117832    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:19.117911 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.117859    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:19.117911 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.117869    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:19.117911 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.117878    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfpgw\" (UniqueName: \"kubernetes.io/projected/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-kube-api-access-bfpgw\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:19.117911 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.117887    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:19.404846 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.404767    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:19.426859 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.426828    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"]
Jul 09 18:26:19.427213 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427191    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50aa2df8-e0a7-4631-b285-cb0ad984ef24" containerName="storage-initializer"
Jul 09 18:26:19.427261 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427217    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="50aa2df8-e0a7-4631-b285-cb0ad984ef24" containerName="storage-initializer"
Jul 09 18:26:19.427261 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427229    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container"
Jul 09 18:26:19.427261 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427237    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container"
Jul 09 18:26:19.427261 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427253    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kube-rbac-proxy"
Jul 09 18:26:19.427383 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427262    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kube-rbac-proxy"
Jul 09 18:26:19.427383 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427277    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="storage-initializer"
Jul 09 18:26:19.427383 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427286    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="storage-initializer"
Jul 09 18:26:19.427383 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427340    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="50aa2df8-e0a7-4631-b285-cb0ad984ef24" containerName="storage-initializer"
Jul 09 18:26:19.427383 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427350    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kserve-container"
Jul 09 18:26:19.427383 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427359    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" containerName="kube-rbac-proxy"
Jul 09 18:26:19.427551 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427424    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50aa2df8-e0a7-4631-b285-cb0ad984ef24" containerName="storage-initializer"
Jul 09 18:26:19.427551 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427432    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="50aa2df8-e0a7-4631-b285-cb0ad984ef24" containerName="storage-initializer"
Jul 09 18:26:19.427551 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.427488    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="50aa2df8-e0a7-4631-b285-cb0ad984ef24" containerName="storage-initializer"
Jul 09 18:26:19.430754 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.430737    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.434304 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.434279    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-70ffd3-kube-rbac-proxy-sar-config\""
Jul 09 18:26:19.434423 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.434282    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-70ffd3\""
Jul 09 18:26:19.434423 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.434281    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-70ffd3-predictor-serving-cert\""
Jul 09 18:26:19.434748 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.434728    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-70ffd3-dockercfg-k8qtc\""
Jul 09 18:26:19.452311 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.452284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"]
Jul 09 18:26:19.520772 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.520738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ggd\" (UniqueName: \"kubernetes.io/projected/78d5605c-0ee4-4395-956d-753d13f745f3-kube-api-access-h9ggd\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.520965 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.520788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.520965 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.520825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78d5605c-0ee4-4395-956d-753d13f745f3-kserve-provision-location\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.520965 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.520864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-cabundle-cert\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.521133 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.520958    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78d5605c-0ee4-4395-956d-753d13f745f3-proxy-tls\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.587403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.587369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:19.622052 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.622021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.622194 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.622060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78d5605c-0ee4-4395-956d-753d13f745f3-kserve-provision-location\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.622194 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.622092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-cabundle-cert\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.622194 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.622127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78d5605c-0ee4-4395-956d-753d13f745f3-proxy-tls\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.622194 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.622148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ggd\" (UniqueName: \"kubernetes.io/projected/78d5605c-0ee4-4395-956d-753d13f745f3-kube-api-access-h9ggd\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.622486 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.622466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78d5605c-0ee4-4395-956d-753d13f745f3-kserve-provision-location\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.622829 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.622810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.622907 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.622809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-cabundle-cert\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.624468 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.624451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78d5605c-0ee4-4395-956d-753d13f745f3-proxy-tls\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.632798 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.632779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ggd\" (UniqueName: \"kubernetes.io/projected/78d5605c-0ee4-4395-956d-753d13f745f3-kube-api-access-h9ggd\") pod \"isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") " pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.741147 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.741108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:19.867787 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.867761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"]
Jul 09 18:26:19.870087 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:26:19.870061    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d5605c_0ee4_4395_956d_753d13f745f3.slice/crio-1066167c7337ed851ede0c177512d7b9c3d4a46043ce4fca07ceee544148fb8f WatchSource:0}: Error finding container 1066167c7337ed851ede0c177512d7b9c3d4a46043ce4fca07ceee544148fb8f: Status 404 returned error can't find the container with id 1066167c7337ed851ede0c177512d7b9c3d4a46043ce4fca07ceee544148fb8f
Jul 09 18:26:19.929812 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.929783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9" event={"ID":"78d5605c-0ee4-4395-956d-753d13f745f3","Type":"ContainerStarted","Data":"018423a8a54d7e0a3ca17dba1a9e449c2faeea214edf7456e9a816a0f32322ae"}
Jul 09 18:26:19.930197 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.929852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9" event={"ID":"78d5605c-0ee4-4395-956d-753d13f745f3","Type":"ContainerStarted","Data":"1066167c7337ed851ede0c177512d7b9c3d4a46043ce4fca07ceee544148fb8f"}
Jul 09 18:26:19.931600 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.931574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q" event={"ID":"a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2","Type":"ContainerDied","Data":"74cfe248b41ea4fd9c68f1ff18fbadb28af3ee21fcfbfbf54286360327926d03"}
Jul 09 18:26:19.931700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.931601    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"
Jul 09 18:26:19.931700 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.931646    2561 scope.go:117] "RemoveContainer" containerID="561e74e5bcf5789fcd6d381e202c652e83bdde071d1c6a9549e1c849d7b57661"
Jul 09 18:26:19.943495 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.943472    2561 scope.go:117] "RemoveContainer" containerID="c5bcce49bb6e15e50cc61a64f1b10febf76b88c55b56aba4b18cb8a25d56823f"
Jul 09 18:26:19.951227 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.951202    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"]
Jul 09 18:26:19.951862 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.951835    2561 scope.go:117] "RemoveContainer" containerID="445705f629b2d3c83ca982067ca7ea0bf161177e04d5dfaf65bd8087f5d0f3cc"
Jul 09 18:26:19.959373 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:19.959354    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-21c109-predictor-7dc6c7b9fb-64n2q"]
Jul 09 18:26:21.313141 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:21.313112    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2" path="/var/lib/kubelet/pods/a2ab2f2e-08ae-4e3e-94c5-6f5ad77812d2/volumes"
Jul 09 18:26:24.951134 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:24.951101    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_78d5605c-0ee4-4395-956d-753d13f745f3/storage-initializer/0.log"
Jul 09 18:26:24.951491 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:24.951138    2561 generic.go:356] "Generic (PLEG): container finished" podID="78d5605c-0ee4-4395-956d-753d13f745f3" containerID="018423a8a54d7e0a3ca17dba1a9e449c2faeea214edf7456e9a816a0f32322ae" exitCode=1
Jul 09 18:26:24.951491 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:24.951177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9" event={"ID":"78d5605c-0ee4-4395-956d-753d13f745f3","Type":"ContainerDied","Data":"018423a8a54d7e0a3ca17dba1a9e449c2faeea214edf7456e9a816a0f32322ae"}
Jul 09 18:26:25.956025 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:25.956000    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_78d5605c-0ee4-4395-956d-753d13f745f3/storage-initializer/0.log"
Jul 09 18:26:25.956433 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:25.956097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9" event={"ID":"78d5605c-0ee4-4395-956d-753d13f745f3","Type":"ContainerStarted","Data":"8537709c97940ce55f1024fe5ba00f01d2b7248102922a8cec8b08f5f1924208"}
Jul 09 18:26:28.968619 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:28.968594    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_78d5605c-0ee4-4395-956d-753d13f745f3/storage-initializer/1.log"
Jul 09 18:26:28.969011 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:28.968995    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_78d5605c-0ee4-4395-956d-753d13f745f3/storage-initializer/0.log"
Jul 09 18:26:28.969188 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:28.969031    2561 generic.go:356] "Generic (PLEG): container finished" podID="78d5605c-0ee4-4395-956d-753d13f745f3" containerID="8537709c97940ce55f1024fe5ba00f01d2b7248102922a8cec8b08f5f1924208" exitCode=1
Jul 09 18:26:28.969188 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:28.969077    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9" event={"ID":"78d5605c-0ee4-4395-956d-753d13f745f3","Type":"ContainerDied","Data":"8537709c97940ce55f1024fe5ba00f01d2b7248102922a8cec8b08f5f1924208"}
Jul 09 18:26:28.969188 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:28.969132    2561 scope.go:117] "RemoveContainer" containerID="018423a8a54d7e0a3ca17dba1a9e449c2faeea214edf7456e9a816a0f32322ae"
Jul 09 18:26:28.969447 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:28.969432    2561 scope.go:117] "RemoveContainer" containerID="018423a8a54d7e0a3ca17dba1a9e449c2faeea214edf7456e9a816a0f32322ae"
Jul 09 18:26:28.979405 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:26:28.979384    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_kserve-ci-e2e-test(78d5605c-0ee4-4395-956d-753d13f745f3)\"" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9" podUID="78d5605c-0ee4-4395-956d-753d13f745f3"
Jul 09 18:26:28.979576 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:26:28.979552    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_kserve-ci-e2e-test_78d5605c-0ee4-4395-956d-753d13f745f3_0 in pod sandbox 1066167c7337ed851ede0c177512d7b9c3d4a46043ce4fca07ceee544148fb8f from index: no such id: '018423a8a54d7e0a3ca17dba1a9e449c2faeea214edf7456e9a816a0f32322ae'" containerID="018423a8a54d7e0a3ca17dba1a9e449c2faeea214edf7456e9a816a0f32322ae"
Jul 09 18:26:28.979643 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:28.979585    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018423a8a54d7e0a3ca17dba1a9e449c2faeea214edf7456e9a816a0f32322ae"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_kserve-ci-e2e-test_78d5605c-0ee4-4395-956d-753d13f745f3_0 in pod sandbox 1066167c7337ed851ede0c177512d7b9c3d4a46043ce4fca07ceee544148fb8f from index: no such id: '018423a8a54d7e0a3ca17dba1a9e449c2faeea214edf7456e9a816a0f32322ae'"
Jul 09 18:26:29.561325 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:29.561292    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"]
Jul 09 18:26:29.603546 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:29.603515    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:29.973701 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:29.973677    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_78d5605c-0ee4-4395-956d-753d13f745f3/storage-initializer/1.log"
Jul 09 18:26:30.098226 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.098206    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_78d5605c-0ee4-4395-956d-753d13f745f3/storage-initializer/1.log"
Jul 09 18:26:30.101714 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.101698    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:30.196275 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.196239    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9ggd\" (UniqueName: \"kubernetes.io/projected/78d5605c-0ee4-4395-956d-753d13f745f3-kube-api-access-h9ggd\") pod \"78d5605c-0ee4-4395-956d-753d13f745f3\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") "
Jul 09 18:26:30.196275 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.196294    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78d5605c-0ee4-4395-956d-753d13f745f3-kserve-provision-location\") pod \"78d5605c-0ee4-4395-956d-753d13f745f3\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") "
Jul 09 18:26:30.196454 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.196316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-kube-rbac-proxy-sar-config\") pod \"78d5605c-0ee4-4395-956d-753d13f745f3\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") "
Jul 09 18:26:30.196454 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.196356    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-cabundle-cert\") pod \"78d5605c-0ee4-4395-956d-753d13f745f3\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") "
Jul 09 18:26:30.196454 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.196381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78d5605c-0ee4-4395-956d-753d13f745f3-proxy-tls\") pod \"78d5605c-0ee4-4395-956d-753d13f745f3\" (UID: \"78d5605c-0ee4-4395-956d-753d13f745f3\") "
Jul 09 18:26:30.196622 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.196538    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d5605c-0ee4-4395-956d-753d13f745f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78d5605c-0ee4-4395-956d-753d13f745f3" (UID: "78d5605c-0ee4-4395-956d-753d13f745f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:30.196710 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.196690    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78d5605c-0ee4-4395-956d-753d13f745f3" (UID: "78d5605c-0ee4-4395-956d-753d13f745f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:30.196710 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.196701    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "78d5605c-0ee4-4395-956d-753d13f745f3" (UID: "78d5605c-0ee4-4395-956d-753d13f745f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:30.198296 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.198268    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d5605c-0ee4-4395-956d-753d13f745f3-kube-api-access-h9ggd" (OuterVolumeSpecName: "kube-api-access-h9ggd") pod "78d5605c-0ee4-4395-956d-753d13f745f3" (UID: "78d5605c-0ee4-4395-956d-753d13f745f3"). InnerVolumeSpecName "kube-api-access-h9ggd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:30.198403 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.198304    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d5605c-0ee4-4395-956d-753d13f745f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78d5605c-0ee4-4395-956d-753d13f745f3" (UID: "78d5605c-0ee4-4395-956d-753d13f745f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:30.297813 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.297729    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9ggd\" (UniqueName: \"kubernetes.io/projected/78d5605c-0ee4-4395-956d-753d13f745f3-kube-api-access-h9ggd\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:30.297813 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.297763    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78d5605c-0ee4-4395-956d-753d13f745f3-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:30.297813 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.297773    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:30.297813 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.297783    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78d5605c-0ee4-4395-956d-753d13f745f3-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:30.297813 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.297792    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78d5605c-0ee4-4395-956d-753d13f745f3-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:26:30.978200 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.978171    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9_78d5605c-0ee4-4395-956d-753d13f745f3/storage-initializer/1.log"
Jul 09 18:26:30.978603 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.978248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9" event={"ID":"78d5605c-0ee4-4395-956d-753d13f745f3","Type":"ContainerDied","Data":"1066167c7337ed851ede0c177512d7b9c3d4a46043ce4fca07ceee544148fb8f"}
Jul 09 18:26:30.978603 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.978273    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"
Jul 09 18:26:30.978603 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:30.978318    2561 scope.go:117] "RemoveContainer" containerID="8537709c97940ce55f1024fe5ba00f01d2b7248102922a8cec8b08f5f1924208"
Jul 09 18:26:31.022201 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:31.022176    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"]
Jul 09 18:26:31.026722 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:31.026701    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-70ffd3-predictor-6b85b8b685-qrkv9"]
Jul 09 18:26:31.312422 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:31.312354    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78d5605c-0ee4-4395-956d-753d13f745f3" path="/var/lib/kubelet/pods/78d5605c-0ee4-4395-956d-753d13f745f3/volumes"
Jul 09 18:26:34.904266 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:34.904205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:34.991046 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:34.991011    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"]
Jul 09 18:26:34.991258 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:34.991245    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d5605c-0ee4-4395-956d-753d13f745f3" containerName="storage-initializer"
Jul 09 18:26:34.991315 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:34.991259    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d5605c-0ee4-4395-956d-753d13f745f3" containerName="storage-initializer"
Jul 09 18:26:34.991315 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:34.991310    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d5605c-0ee4-4395-956d-753d13f745f3" containerName="storage-initializer"
Jul 09 18:26:34.991394 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:34.991350    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d5605c-0ee4-4395-956d-753d13f745f3" containerName="storage-initializer"
Jul 09 18:26:34.991394 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:34.991356    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d5605c-0ee4-4395-956d-753d13f745f3" containerName="storage-initializer"
Jul 09 18:26:34.991464 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:34.991400    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d5605c-0ee4-4395-956d-753d13f745f3" containerName="storage-initializer"
Jul 09 18:26:34.995538 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:34.995519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.005218 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.005199    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:26:35.005323 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.005281    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:26:35.006141 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.006127    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-c2694-kube-rbac-proxy-sar-config\""
Jul 09 18:26:35.006235 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.006155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:26:35.007849 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.007835    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-c2694-predictor-serving-cert\""
Jul 09 18:26:35.010761 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.010742    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:26:35.010878 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.010859    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tfp25\""
Jul 09 18:26:35.012136 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.012119    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"]
Jul 09 18:26:35.028848 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.028830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:35.029857 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.029835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/456d9894-0764-4a25-a512-2d2458f1bd46-proxy-tls\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.029960 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.029877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/456d9894-0764-4a25-a512-2d2458f1bd46-kserve-provision-location\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.030030 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.029966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.030081 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.030027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-cabundle-cert\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.030123 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.030088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gffzk\" (UniqueName: \"kubernetes.io/projected/456d9894-0764-4a25-a512-2d2458f1bd46-kube-api-access-gffzk\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.130785 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.130759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.130873 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.130804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-cabundle-cert\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.130873 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.130828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gffzk\" (UniqueName: \"kubernetes.io/projected/456d9894-0764-4a25-a512-2d2458f1bd46-kube-api-access-gffzk\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.131013 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.130941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/456d9894-0764-4a25-a512-2d2458f1bd46-proxy-tls\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.131013 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.131004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/456d9894-0764-4a25-a512-2d2458f1bd46-kserve-provision-location\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.131346 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.131326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/456d9894-0764-4a25-a512-2d2458f1bd46-kserve-provision-location\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.131414 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.131397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-cabundle-cert\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.131509 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.131489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.133398 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.133379    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/456d9894-0764-4a25-a512-2d2458f1bd46-proxy-tls\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.147118 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.147093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gffzk\" (UniqueName: \"kubernetes.io/projected/456d9894-0764-4a25-a512-2d2458f1bd46-kube-api-access-gffzk\") pod \"raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") " pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.301255 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.301221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:35.305144 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.305126    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:35.452777 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.452749    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"]
Jul 09 18:26:35.455798 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:26:35.455768    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456d9894_0764_4a25_a512_2d2458f1bd46.slice/crio-e69e852dd7fb83288555a21c026aac5527fe262d77b69a2328c0c7ca71ef67ef WatchSource:0}: Error finding container e69e852dd7fb83288555a21c026aac5527fe262d77b69a2328c0c7ca71ef67ef: Status 404 returned error can't find the container with id e69e852dd7fb83288555a21c026aac5527fe262d77b69a2328c0c7ca71ef67ef
Jul 09 18:26:35.997971 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.997917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" event={"ID":"456d9894-0764-4a25-a512-2d2458f1bd46","Type":"ContainerStarted","Data":"0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24"}
Jul 09 18:26:35.998328 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:35.997994    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" event={"ID":"456d9894-0764-4a25-a512-2d2458f1bd46","Type":"ContainerStarted","Data":"e69e852dd7fb83288555a21c026aac5527fe262d77b69a2328c0c7ca71ef67ef"}
Jul 09 18:26:40.013433 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:40.013391    2561 generic.go:356] "Generic (PLEG): container finished" podID="456d9894-0764-4a25-a512-2d2458f1bd46" containerID="0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24" exitCode=0
Jul 09 18:26:40.013910 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:40.013473    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" event={"ID":"456d9894-0764-4a25-a512-2d2458f1bd46","Type":"ContainerDied","Data":"0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24"}
Jul 09 18:26:41.018962 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:41.018913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" event={"ID":"456d9894-0764-4a25-a512-2d2458f1bd46","Type":"ContainerStarted","Data":"20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b"}
Jul 09 18:26:41.019309 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:41.018991    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" event={"ID":"456d9894-0764-4a25-a512-2d2458f1bd46","Type":"ContainerStarted","Data":"c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b"}
Jul 09 18:26:41.019309 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:41.019167    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:41.040628 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:41.040588    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podStartSLOduration=3.221014549 podStartE2EDuration="7.040574279s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81955973s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:41.038857083 +0000 UTC m=+1322.365332806" watchObservedRunningTime="2026-07-09 18:26:41.040574279 +0000 UTC m=+1322.367049987"
Jul 09 18:26:42.023185 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:42.023148    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:42.024353 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:42.024326    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:26:43.027173 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:43.027129    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:26:48.031927 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:48.031899    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:26:48.032425 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:48.032400    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:26:58.033200 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:26:58.033160    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:27:08.032680 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:27:08.032594    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:27:14.371070 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:27:14.371037    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:27:18.033187 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:27:18.033148    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:27:28.032526 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:27:28.032488    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:27:38.033136 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:27:38.033100    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:27:48.032777 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:27:48.032734    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:27:58.033588 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:27:58.033556    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:28:04.857444 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:04.857414    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"]
Jul 09 18:28:04.857846 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:04.857646    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" containerID="cri-o://c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b" gracePeriod=30
Jul 09 18:28:04.857846 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:04.857705    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kube-rbac-proxy" containerID="cri-o://20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b" gracePeriod=30
Jul 09 18:28:04.878830 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:04.878786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:05.308673 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:05.308642    2561 generic.go:356] "Generic (PLEG): container finished" podID="456d9894-0764-4a25-a512-2d2458f1bd46" containerID="20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b" exitCode=2
Jul 09 18:28:05.312035 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:05.312011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" event={"ID":"456d9894-0764-4a25-a512-2d2458f1bd46","Type":"ContainerDied","Data":"20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b"}
Jul 09 18:28:08.027851 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:08.027805    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 09 18:28:08.033153 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:08.033130    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 09 18:28:09.194371 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.194347    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:28:09.289856 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.289817    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-kube-rbac-proxy-sar-config\") pod \"456d9894-0764-4a25-a512-2d2458f1bd46\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") "
Jul 09 18:28:09.290053 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.289869    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/456d9894-0764-4a25-a512-2d2458f1bd46-proxy-tls\") pod \"456d9894-0764-4a25-a512-2d2458f1bd46\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") "
Jul 09 18:28:09.290053 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.289884    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-cabundle-cert\") pod \"456d9894-0764-4a25-a512-2d2458f1bd46\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") "
Jul 09 18:28:09.290053 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.289909    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/456d9894-0764-4a25-a512-2d2458f1bd46-kserve-provision-location\") pod \"456d9894-0764-4a25-a512-2d2458f1bd46\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") "
Jul 09 18:28:09.290190 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.290062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gffzk\" (UniqueName: \"kubernetes.io/projected/456d9894-0764-4a25-a512-2d2458f1bd46-kube-api-access-gffzk\") pod \"456d9894-0764-4a25-a512-2d2458f1bd46\" (UID: \"456d9894-0764-4a25-a512-2d2458f1bd46\") "
Jul 09 18:28:09.290307 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.290265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456d9894-0764-4a25-a512-2d2458f1bd46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "456d9894-0764-4a25-a512-2d2458f1bd46" (UID: "456d9894-0764-4a25-a512-2d2458f1bd46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:09.290307 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.290289    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "456d9894-0764-4a25-a512-2d2458f1bd46" (UID: "456d9894-0764-4a25-a512-2d2458f1bd46"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:09.290430 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.290342    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "456d9894-0764-4a25-a512-2d2458f1bd46" (UID: "456d9894-0764-4a25-a512-2d2458f1bd46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:09.292128 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.292064    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456d9894-0764-4a25-a512-2d2458f1bd46-kube-api-access-gffzk" (OuterVolumeSpecName: "kube-api-access-gffzk") pod "456d9894-0764-4a25-a512-2d2458f1bd46" (UID: "456d9894-0764-4a25-a512-2d2458f1bd46"). InnerVolumeSpecName "kube-api-access-gffzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:09.292128 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.292074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d9894-0764-4a25-a512-2d2458f1bd46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "456d9894-0764-4a25-a512-2d2458f1bd46" (UID: "456d9894-0764-4a25-a512-2d2458f1bd46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:09.322958 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.322924    2561 generic.go:356] "Generic (PLEG): container finished" podID="456d9894-0764-4a25-a512-2d2458f1bd46" containerID="c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b" exitCode=0
Jul 09 18:28:09.323057 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.322971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" event={"ID":"456d9894-0764-4a25-a512-2d2458f1bd46","Type":"ContainerDied","Data":"c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b"}
Jul 09 18:28:09.323057 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.322989    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"
Jul 09 18:28:09.323057 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.323022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4" event={"ID":"456d9894-0764-4a25-a512-2d2458f1bd46","Type":"ContainerDied","Data":"e69e852dd7fb83288555a21c026aac5527fe262d77b69a2328c0c7ca71ef67ef"}
Jul 09 18:28:09.323057 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.323048    2561 scope.go:117] "RemoveContainer" containerID="20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b"
Jul 09 18:28:09.330880 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.330863    2561 scope.go:117] "RemoveContainer" containerID="c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b"
Jul 09 18:28:09.337488 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.337468    2561 scope.go:117] "RemoveContainer" containerID="0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24"
Jul 09 18:28:09.344153 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.344129    2561 scope.go:117] "RemoveContainer" containerID="20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b"
Jul 09 18:28:09.344397 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:28:09.344380    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b\": container with ID starting with 20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b not found: ID does not exist" containerID="20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b"
Jul 09 18:28:09.344440 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.344403    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b"} err="failed to get container status \"20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b\": rpc error: code = NotFound desc = could not find container \"20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b\": container with ID starting with 20f0625e9f87b6879c4cc52309dc2a51f40acd05c9237e3eb91e6c33dfc6be6b not found: ID does not exist"
Jul 09 18:28:09.344440 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.344421    2561 scope.go:117] "RemoveContainer" containerID="c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b"
Jul 09 18:28:09.344639 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:28:09.344622    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b\": container with ID starting with c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b not found: ID does not exist" containerID="c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b"
Jul 09 18:28:09.344690 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.344641    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b"} err="failed to get container status \"c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b\": rpc error: code = NotFound desc = could not find container \"c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b\": container with ID starting with c1db9948dcf731b707daa1e86ecdfe0ec557c365ef959ef82036056a2fda165b not found: ID does not exist"
Jul 09 18:28:09.344690 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.344652    2561 scope.go:117] "RemoveContainer" containerID="0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24"
Jul 09 18:28:09.344862 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:28:09.344850    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24\": container with ID starting with 0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24 not found: ID does not exist" containerID="0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24"
Jul 09 18:28:09.344900 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.344864    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24"} err="failed to get container status \"0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24\": rpc error: code = NotFound desc = could not find container \"0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24\": container with ID starting with 0a8257c01e88f8870d3d23ffd64309f0a892241b4f0e8aa7b6b5c42ba1975c24 not found: ID does not exist"
Jul 09 18:28:09.350648 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.350629    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"]
Jul 09 18:28:09.361594 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.361577    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-c2694-predictor-6cb6b7896f-kp8n4"]
Jul 09 18:28:09.390995 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.390976    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:28:09.390995 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.390992    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/456d9894-0764-4a25-a512-2d2458f1bd46-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:28:09.391123 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.391001    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/456d9894-0764-4a25-a512-2d2458f1bd46-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:28:09.391123 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.391010    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/456d9894-0764-4a25-a512-2d2458f1bd46-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:28:09.391123 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:09.391018    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gffzk\" (UniqueName: \"kubernetes.io/projected/456d9894-0764-4a25-a512-2d2458f1bd46-kube-api-access-gffzk\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:28:10.055828 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.055797    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:10.133542 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.133509    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"]
Jul 09 18:28:10.133767 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.133753    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container"
Jul 09 18:28:10.133767 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.133768    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container"
Jul 09 18:28:10.133908 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.133780    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kube-rbac-proxy"
Jul 09 18:28:10.133908 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.133787    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kube-rbac-proxy"
Jul 09 18:28:10.133908 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.133802    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="storage-initializer"
Jul 09 18:28:10.133908 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.133808    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="storage-initializer"
Jul 09 18:28:10.133908 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.133850    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kserve-container"
Jul 09 18:28:10.133908 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.133858    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" containerName="kube-rbac-proxy"
Jul 09 18:28:10.138205 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.138185    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.140932 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.140909    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:28:10.140932 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.140929    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:28:10.141214 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.141200    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:28:10.141298 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.141216    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-a83a2-kube-rbac-proxy-sar-config\""
Jul 09 18:28:10.141628 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.141613    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-a83a2-predictor-serving-cert\""
Jul 09 18:28:10.141628 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.141622    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tfp25\""
Jul 09 18:28:10.141749 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.141665    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:28:10.150620 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.150601    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"]
Jul 09 18:28:10.196059 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.196031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f43d3c-47b4-43d2-b87b-7ef17af9253d-proxy-tls\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.196410 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.196061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.196410 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.196082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8dnl\" (UniqueName: \"kubernetes.io/projected/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-api-access-h8dnl\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.196410 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.196155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kserve-provision-location\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.196410 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.196180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-cabundle-cert\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.296562 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.296536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f43d3c-47b4-43d2-b87b-7ef17af9253d-proxy-tls\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.296677 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.296567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.296677 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.296589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8dnl\" (UniqueName: \"kubernetes.io/projected/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-api-access-h8dnl\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.296677 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.296627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kserve-provision-location\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.296677 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.296654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-cabundle-cert\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.296894 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:28:10.296688    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-serving-cert: secret "raw-sklearn-runtime-a83a2-predictor-serving-cert" not found
Jul 09 18:28:10.296894 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:28:10.296753    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f43d3c-47b4-43d2-b87b-7ef17af9253d-proxy-tls podName:06f43d3c-47b4-43d2-b87b-7ef17af9253d nodeName:}" failed. No retries permitted until 2026-07-09 18:28:10.79673175 +0000 UTC m=+1412.123207440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/06f43d3c-47b4-43d2-b87b-7ef17af9253d-proxy-tls") pod "raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" (UID: "06f43d3c-47b4-43d2-b87b-7ef17af9253d") : secret "raw-sklearn-runtime-a83a2-predictor-serving-cert" not found
Jul 09 18:28:10.297081 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.297065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kserve-provision-location\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.297385 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.297365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-cabundle-cert\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.297440 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.297365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.308926 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.308878    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8dnl\" (UniqueName: \"kubernetes.io/projected/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-api-access-h8dnl\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.410511 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.410470    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:10.800414 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.800369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f43d3c-47b4-43d2-b87b-7ef17af9253d-proxy-tls\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:10.802728 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:10.802709    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f43d3c-47b4-43d2-b87b-7ef17af9253d-proxy-tls\") pod \"raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:11.050882 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:11.050800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:11.179267 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:11.179244    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"]
Jul 09 18:28:11.181140 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:28:11.181109    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f43d3c_47b4_43d2_b87b_7ef17af9253d.slice/crio-7bf341c1a72af10fce99005e8144ca538fb3e69deb5e40a4aa0e3648508bd2bd WatchSource:0}: Error finding container 7bf341c1a72af10fce99005e8144ca538fb3e69deb5e40a4aa0e3648508bd2bd: Status 404 returned error can't find the container with id 7bf341c1a72af10fce99005e8144ca538fb3e69deb5e40a4aa0e3648508bd2bd
Jul 09 18:28:11.316131 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:11.315693    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="456d9894-0764-4a25-a512-2d2458f1bd46" path="/var/lib/kubelet/pods/456d9894-0764-4a25-a512-2d2458f1bd46/volumes"
Jul 09 18:28:11.331577 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:11.331535    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" event={"ID":"06f43d3c-47b4-43d2-b87b-7ef17af9253d","Type":"ContainerStarted","Data":"e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103"}
Jul 09 18:28:11.331713 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:11.331615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" event={"ID":"06f43d3c-47b4-43d2-b87b-7ef17af9253d","Type":"ContainerStarted","Data":"7bf341c1a72af10fce99005e8144ca538fb3e69deb5e40a4aa0e3648508bd2bd"}
Jul 09 18:28:15.346201 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:15.346116    2561 generic.go:356] "Generic (PLEG): container finished" podID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerID="e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103" exitCode=0
Jul 09 18:28:15.346201 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:15.346185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" event={"ID":"06f43d3c-47b4-43d2-b87b-7ef17af9253d","Type":"ContainerDied","Data":"e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103"}
Jul 09 18:28:16.351920 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:16.351885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" event={"ID":"06f43d3c-47b4-43d2-b87b-7ef17af9253d","Type":"ContainerStarted","Data":"0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22"}
Jul 09 18:28:16.352309 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:16.351956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" event={"ID":"06f43d3c-47b4-43d2-b87b-7ef17af9253d","Type":"ContainerStarted","Data":"3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c"}
Jul 09 18:28:16.352309 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:16.352189    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:16.375522 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:16.375480    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podStartSLOduration=2.559476615 podStartE2EDuration="6.375464216s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.815987601s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:16.374085017 +0000 UTC m=+1417.700560724" watchObservedRunningTime="2026-07-09 18:28:16.375464216 +0000 UTC m=+1417.701939925"
Jul 09 18:28:17.355071 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:17.355037    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:17.356475 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:17.356444    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:28:18.358297 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:18.358263    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:28:23.363224 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:23.363193    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:28:23.363626 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:23.363602    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:28:33.364677 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:33.364586    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:28:43.363873 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:43.363833    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:28:53.363984 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:28:53.363926    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:29:01.921285 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:01.921248    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:01.993893 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:01.993860    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:03.364331 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:03.364291    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:29:09.422121 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:09.422087    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:13.364296 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:13.364260    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:29:23.363826 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:23.363788    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:29:33.364818 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:33.364789    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:29:39.295919 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:39.295893    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:29:39.296439 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:39.296423    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:29:40.229750 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:40.229717    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"]
Jul 09 18:29:40.230057 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:40.230031    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" containerID="cri-o://3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c" gracePeriod=30
Jul 09 18:29:40.230176 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:40.230120    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kube-rbac-proxy" containerID="cri-o://0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22" gracePeriod=30
Jul 09 18:29:40.261725 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:40.261704    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:40.641343 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:40.641238    2561 generic.go:356] "Generic (PLEG): container finished" podID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerID="0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22" exitCode=2
Jul 09 18:29:40.641759 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:40.641320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" event={"ID":"06f43d3c-47b4-43d2-b87b-7ef17af9253d","Type":"ContainerDied","Data":"0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22"}
Jul 09 18:29:43.359008 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:43.358972    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 09 18:29:43.364282 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:43.364258    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:29:44.472305 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.472284    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:29:44.560110 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.560043    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-rbac-proxy-sar-config\") pod \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") "
Jul 09 18:29:44.560110 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.560076    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-cabundle-cert\") pod \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") "
Jul 09 18:29:44.560110 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.560106    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kserve-provision-location\") pod \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") "
Jul 09 18:29:44.560360 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.560138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f43d3c-47b4-43d2-b87b-7ef17af9253d-proxy-tls\") pod \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") "
Jul 09 18:29:44.560360 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.560158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8dnl\" (UniqueName: \"kubernetes.io/projected/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-api-access-h8dnl\") pod \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\" (UID: \"06f43d3c-47b4-43d2-b87b-7ef17af9253d\") "
Jul 09 18:29:44.560465 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.560380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06f43d3c-47b4-43d2-b87b-7ef17af9253d" (UID: "06f43d3c-47b4-43d2-b87b-7ef17af9253d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:29:44.560465 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.560449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06f43d3c-47b4-43d2-b87b-7ef17af9253d" (UID: "06f43d3c-47b4-43d2-b87b-7ef17af9253d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:29:44.560574 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.560555    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06f43d3c-47b4-43d2-b87b-7ef17af9253d" (UID: "06f43d3c-47b4-43d2-b87b-7ef17af9253d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:29:44.562147 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.562127    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-api-access-h8dnl" (OuterVolumeSpecName: "kube-api-access-h8dnl") pod "06f43d3c-47b4-43d2-b87b-7ef17af9253d" (UID: "06f43d3c-47b4-43d2-b87b-7ef17af9253d"). InnerVolumeSpecName "kube-api-access-h8dnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:29:44.562255 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.562174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f43d3c-47b4-43d2-b87b-7ef17af9253d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06f43d3c-47b4-43d2-b87b-7ef17af9253d" (UID: "06f43d3c-47b4-43d2-b87b-7ef17af9253d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:29:44.657429 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.657401    2561 generic.go:356] "Generic (PLEG): container finished" podID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerID="3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c" exitCode=0
Jul 09 18:29:44.657552 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.657461    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"
Jul 09 18:29:44.657552 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.657464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" event={"ID":"06f43d3c-47b4-43d2-b87b-7ef17af9253d","Type":"ContainerDied","Data":"3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c"}
Jul 09 18:29:44.657620 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.657601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9" event={"ID":"06f43d3c-47b4-43d2-b87b-7ef17af9253d","Type":"ContainerDied","Data":"7bf341c1a72af10fce99005e8144ca538fb3e69deb5e40a4aa0e3648508bd2bd"}
Jul 09 18:29:44.657659 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.657647    2561 scope.go:117] "RemoveContainer" containerID="0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22"
Jul 09 18:29:44.660526 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.660507    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kserve-provision-location\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:29:44.660526 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.660527    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06f43d3c-47b4-43d2-b87b-7ef17af9253d-proxy-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:29:44.660662 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.660537    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8dnl\" (UniqueName: \"kubernetes.io/projected/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-api-access-h8dnl\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:29:44.660662 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.660546    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:29:44.660662 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.660555    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06f43d3c-47b4-43d2-b87b-7ef17af9253d-cabundle-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Jul 09 18:29:44.665557 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.665544    2561 scope.go:117] "RemoveContainer" containerID="3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c"
Jul 09 18:29:44.672077 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.672056    2561 scope.go:117] "RemoveContainer" containerID="e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103"
Jul 09 18:29:44.677683 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.677662    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"]
Jul 09 18:29:44.679060 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.679045    2561 scope.go:117] "RemoveContainer" containerID="0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22"
Jul 09 18:29:44.679293 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:29:44.679277    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22\": container with ID starting with 0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22 not found: ID does not exist" containerID="0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22"
Jul 09 18:29:44.679359 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.679301    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22"} err="failed to get container status \"0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22\": rpc error: code = NotFound desc = could not find container \"0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22\": container with ID starting with 0d05da45d5d11bd40d0cb25de9a549914ebd10508c89d92329db0a0ca05a9b22 not found: ID does not exist"
Jul 09 18:29:44.679359 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.679319    2561 scope.go:117] "RemoveContainer" containerID="3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c"
Jul 09 18:29:44.679560 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:29:44.679541    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c\": container with ID starting with 3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c not found: ID does not exist" containerID="3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c"
Jul 09 18:29:44.679605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.679566    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c"} err="failed to get container status \"3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c\": rpc error: code = NotFound desc = could not find container \"3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c\": container with ID starting with 3fbdf44040ec1484d82223c8465d7221c4c163002b58e14a9d3255f2c719be6c not found: ID does not exist"
Jul 09 18:29:44.679605 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.679582    2561 scope.go:117] "RemoveContainer" containerID="e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103"
Jul 09 18:29:44.679800 ip-10-0-140-229 kubenswrapper[2561]: E0709 18:29:44.679786    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103\": container with ID starting with e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103 not found: ID does not exist" containerID="e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103"
Jul 09 18:29:44.679836 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.679802    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103"} err="failed to get container status \"e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103\": rpc error: code = NotFound desc = could not find container \"e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103\": container with ID starting with e7fc793df1b34d96e4a96795310720f08c96a17da61ad978379cb1c469a7a103 not found: ID does not exist"
Jul 09 18:29:44.683434 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:44.683413    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-a83a2-predictor-ddcccd765-2sxf9"]
Jul 09 18:29:45.313742 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:45.313713    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" path="/var/lib/kubelet/pods/06f43d3c-47b4-43d2-b87b-7ef17af9253d/volumes"
Jul 09 18:29:45.958259 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:45.958233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:46.043122 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:29:46.043076    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:12.330989 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.330848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:12.452005 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.451967    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cf7pf/must-gather-dcjvv"]
Jul 09 18:30:12.452209 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.452198    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container"
Jul 09 18:30:12.452253 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.452210    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container"
Jul 09 18:30:12.452253 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.452222    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="storage-initializer"
Jul 09 18:30:12.452253 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.452228    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="storage-initializer"
Jul 09 18:30:12.452253 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.452239    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kube-rbac-proxy"
Jul 09 18:30:12.452253 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.452244    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kube-rbac-proxy"
Jul 09 18:30:12.452411 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.452282    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kserve-container"
Jul 09 18:30:12.452411 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.452292    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f43d3c-47b4-43d2-b87b-7ef17af9253d" containerName="kube-rbac-proxy"
Jul 09 18:30:12.455016 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.454994    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cf7pf/must-gather-dcjvv"
Jul 09 18:30:12.457465 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.457441    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cf7pf\"/\"default-dockercfg-4l96w\""
Jul 09 18:30:12.457595 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.457516    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cf7pf\"/\"openshift-service-ca.crt\""
Jul 09 18:30:12.457809 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.457791    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cf7pf\"/\"kube-root-ca.crt\""
Jul 09 18:30:12.470840 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.470820    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cf7pf/must-gather-dcjvv"]
Jul 09 18:30:12.472369 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.472350    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:12.564204 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.564175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f819539a-fa4a-44d6-9a12-84b21a821b3f-must-gather-output\") pod \"must-gather-dcjvv\" (UID: \"f819539a-fa4a-44d6-9a12-84b21a821b3f\") " pod="openshift-must-gather-cf7pf/must-gather-dcjvv"
Jul 09 18:30:12.564368 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.564223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7jqb\" (UniqueName: \"kubernetes.io/projected/f819539a-fa4a-44d6-9a12-84b21a821b3f-kube-api-access-h7jqb\") pod \"must-gather-dcjvv\" (UID: \"f819539a-fa4a-44d6-9a12-84b21a821b3f\") " pod="openshift-must-gather-cf7pf/must-gather-dcjvv"
Jul 09 18:30:12.567690 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.567670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:12.665564 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.665489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f819539a-fa4a-44d6-9a12-84b21a821b3f-must-gather-output\") pod \"must-gather-dcjvv\" (UID: \"f819539a-fa4a-44d6-9a12-84b21a821b3f\") " pod="openshift-must-gather-cf7pf/must-gather-dcjvv"
Jul 09 18:30:12.665564 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.665547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7jqb\" (UniqueName: \"kubernetes.io/projected/f819539a-fa4a-44d6-9a12-84b21a821b3f-kube-api-access-h7jqb\") pod \"must-gather-dcjvv\" (UID: \"f819539a-fa4a-44d6-9a12-84b21a821b3f\") " pod="openshift-must-gather-cf7pf/must-gather-dcjvv"
Jul 09 18:30:12.665810 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.665793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f819539a-fa4a-44d6-9a12-84b21a821b3f-must-gather-output\") pod \"must-gather-dcjvv\" (UID: \"f819539a-fa4a-44d6-9a12-84b21a821b3f\") " pod="openshift-must-gather-cf7pf/must-gather-dcjvv"
Jul 09 18:30:12.674616 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.674598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7jqb\" (UniqueName: \"kubernetes.io/projected/f819539a-fa4a-44d6-9a12-84b21a821b3f-kube-api-access-h7jqb\") pod \"must-gather-dcjvv\" (UID: \"f819539a-fa4a-44d6-9a12-84b21a821b3f\") " pod="openshift-must-gather-cf7pf/must-gather-dcjvv"
Jul 09 18:30:12.764202 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.764169    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cf7pf/must-gather-dcjvv"
Jul 09 18:30:12.891157 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.891126    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cf7pf/must-gather-dcjvv"]
Jul 09 18:30:12.894700 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:30:12.894673    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf819539a_fa4a_44d6_9a12_84b21a821b3f.slice/crio-265f1b1b9d114eca813696e5a9904eff83d7a8dd503b2ccbbf70c3658d6476b7 WatchSource:0}: Error finding container 265f1b1b9d114eca813696e5a9904eff83d7a8dd503b2ccbbf70c3658d6476b7: Status 404 returned error can't find the container with id 265f1b1b9d114eca813696e5a9904eff83d7a8dd503b2ccbbf70c3658d6476b7
Jul 09 18:30:12.896545 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:12.896528    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:30:13.754079 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:13.754036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cf7pf/must-gather-dcjvv" event={"ID":"f819539a-fa4a-44d6-9a12-84b21a821b3f","Type":"ContainerStarted","Data":"265f1b1b9d114eca813696e5a9904eff83d7a8dd503b2ccbbf70c3658d6476b7"}
Jul 09 18:30:14.763803 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:14.763766    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cf7pf/must-gather-dcjvv" event={"ID":"f819539a-fa4a-44d6-9a12-84b21a821b3f","Type":"ContainerStarted","Data":"295168fcbc4a89c97e1dbcfe6ce30f0f4423eff5d9a4a215557cb5a87a969700"}
Jul 09 18:30:14.764262 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:14.763817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cf7pf/must-gather-dcjvv" event={"ID":"f819539a-fa4a-44d6-9a12-84b21a821b3f","Type":"ContainerStarted","Data":"8c21c850c3d389aeaf71f6f152b584b711b03f9b86c457dcf90147fdeeace77e"}
Jul 09 18:30:14.781341 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:14.781279    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cf7pf/must-gather-dcjvv" podStartSLOduration=1.831712952 podStartE2EDuration="2.781250769s" totalImagesPullingTime="949.537817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:14.780269556 +0000 UTC m=+1536.106745269" watchObservedRunningTime="2026-07-09 18:30:14.781250769 +0000 UTC m=+1536.107726480"
Jul 09 18:30:15.317656 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:15.317625    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fxrfv_5d52ad56-9937-4f10-88c3-051fe2f35027/global-pull-secret-syncer/0.log"
Jul 09 18:30:15.484996 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:15.484958    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log"
Jul 09 18:30:15.623220 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:15.623135    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-229.ec2.internal_641e9bded5c7fc9e1b94316361d7f097/haproxy/0.log"
Jul 09 18:30:18.394811 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:18.394773    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:18.471440 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:18.471371    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25l7d_6cab92a5-eb6e-4d21-b073-f4909c29eee7/konnectivity-agent/0.log" limit=1024
Jul 09 18:30:19.945337 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:19.945312    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v9lvc_5a180287-6787-48f3-91d6-4c1ffe89df3a/node-exporter/0.log"
Jul 09 18:30:19.975562 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:19.975538    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v9lvc_5a180287-6787-48f3-91d6-4c1ffe89df3a/kube-rbac-proxy/0.log"
Jul 09 18:30:20.002351 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:20.002324    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v9lvc_5a180287-6787-48f3-91d6-4c1ffe89df3a/init-textfile/0.log"
Jul 09 18:30:23.701154 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.701114    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"]
Jul 09 18:30:23.705618 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.705593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.723806 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.723779    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"]
Jul 09 18:30:23.761363 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.761318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-podres\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.761548 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.761391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b9mw\" (UniqueName: \"kubernetes.io/projected/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-kube-api-access-6b9mw\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.761548 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.761444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-lib-modules\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.761548 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.761497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-proc\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.761548 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.761524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-sys\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.862410 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.862372    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-lib-modules\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.862598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.862419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-proc\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.862598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.862437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-sys\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.862598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.862459    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-podres\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.862598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.862496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b9mw\" (UniqueName: \"kubernetes.io/projected/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-kube-api-access-6b9mw\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.862598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.862525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-proc\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.862598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.862525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-sys\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.862598 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.862575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-lib-modules\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.862880 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.862604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-podres\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:23.872282 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:23.872259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b9mw\" (UniqueName: \"kubernetes.io/projected/06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c-kube-api-access-6b9mw\") pod \"perf-node-gather-daemonset-l7gtz\" (UID: \"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c\") " pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:24.016726 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:24.016639    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:24.179575 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:24.179507    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"]
Jul 09 18:30:24.182498 ip-10-0-140-229 kubenswrapper[2561]: W0709 18:30:24.182470    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod06bd06d7_ae61_4cf2_b86f_cd9b6826fb4c.slice/crio-f0b780d94d15cd11efe1c80427d9b8b5db48c515f19304db2042e2050998111f WatchSource:0}: Error finding container f0b780d94d15cd11efe1c80427d9b8b5db48c515f19304db2042e2050998111f: Status 404 returned error can't find the container with id f0b780d94d15cd11efe1c80427d9b8b5db48c515f19304db2042e2050998111f
Jul 09 18:30:24.636666 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:24.636631    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9f5n4_d5ccc282-8bfa-47dc-80b1-13c7232c51ab/dns/0.log"
Jul 09 18:30:24.662624 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:24.662595    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9f5n4_d5ccc282-8bfa-47dc-80b1-13c7232c51ab/kube-rbac-proxy/0.log"
Jul 09 18:30:24.769578 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:24.769538    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxxk7_e1b578cd-ec3f-4c6c-8acb-7fe9818c4bf8/dns-node-resolver/0.log"
Jul 09 18:30:24.813760 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:24.813727    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz" event={"ID":"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c","Type":"ContainerStarted","Data":"2cd37501d71dbe20c554f2b7f64ca3c30b1a69dcb3cdcf263189038edf8ddf99"}
Jul 09 18:30:24.813992 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:24.813777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz" event={"ID":"06bd06d7-ae61-4cf2-b86f-cd9b6826fb4c","Type":"ContainerStarted","Data":"f0b780d94d15cd11efe1c80427d9b8b5db48c515f19304db2042e2050998111f"}
Jul 09 18:30:24.813992 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:24.813881    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:24.830401 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:24.830336    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz" podStartSLOduration=1.830315631 podStartE2EDuration="1.830315631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:24.829202444 +0000 UTC m=+1546.155678166" watchObservedRunningTime="2026-07-09 18:30:24.830315631 +0000 UTC m=+1546.156791342"
Jul 09 18:30:25.276226 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:25.276200    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6666f97c54-7dng9_71500497-abe2-44b4-afd0-e956f18e2d01/registry/0.log"
Jul 09 18:30:25.297286 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:25.297247    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5jhnd_9ec2f0e2-e620-4920-8313-1a3107dca62a/node-ca/0.log"
Jul 09 18:30:26.590936 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:26.590910    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wg2jq_9c23fe5b-f395-4f2d-9304-52e9d6adce51/serve-healthcheck-canary/0.log"
Jul 09 18:30:27.058857 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:27.058827    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gx8b2_03a22790-e419-40d7-abdb-6517d9103e1c/kube-rbac-proxy/0.log"
Jul 09 18:30:27.081793 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:27.081766    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gx8b2_03a22790-e419-40d7-abdb-6517d9103e1c/exporter/0.log"
Jul 09 18:30:27.106564 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:27.106540    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gx8b2_03a22790-e419-40d7-abdb-6517d9103e1c/extractor/0.log"
Jul 09 18:30:29.527251 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:29.527216    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-wxbr6_0c3aceb9-eb87-4559-9336-c1fbf150f532/manager/0.log"
Jul 09 18:30:30.828161 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:30.828133    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cf7pf/perf-node-gather-daemonset-l7gtz"
Jul 09 18:30:35.133834 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.133674    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7fb5b_bcbf8808-7608-4153-90f4-2dcef7231073/kube-multus/0.log"
Jul 09 18:30:35.197528 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.197500    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4zdxd_a3555656-954a-49c5-a49c-50c44b2f7ed0/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:30:35.220735 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.220717    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4zdxd_a3555656-954a-49c5-a49c-50c44b2f7ed0/egress-router-binary-copy/0.log"
Jul 09 18:30:35.242996 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.242971    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4zdxd_a3555656-954a-49c5-a49c-50c44b2f7ed0/cni-plugins/0.log"
Jul 09 18:30:35.265040 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.265020    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4zdxd_a3555656-954a-49c5-a49c-50c44b2f7ed0/bond-cni-plugin/0.log"
Jul 09 18:30:35.287694 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.287667    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4zdxd_a3555656-954a-49c5-a49c-50c44b2f7ed0/routeoverride-cni/0.log"
Jul 09 18:30:35.318237 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.318214    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4zdxd_a3555656-954a-49c5-a49c-50c44b2f7ed0/whereabouts-cni-bincopy/0.log"
Jul 09 18:30:35.341892 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.341872    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4zdxd_a3555656-954a-49c5-a49c-50c44b2f7ed0/whereabouts-cni/0.log"
Jul 09 18:30:35.863239 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.863208    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l5m2b_641493c8-16d5-46c0-bd0a-8763e005dcdf/network-metrics-daemon/0.log"
Jul 09 18:30:35.889329 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:35.889304    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l5m2b_641493c8-16d5-46c0-bd0a-8763e005dcdf/kube-rbac-proxy/0.log"
Jul 09 18:30:36.882558 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:36.882528    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-controller/0.log"
Jul 09 18:30:36.903957 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:36.903927    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/1.log"
Jul 09 18:30:36.904071 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:36.904026    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovn-acl-logging/0.log"
Jul 09 18:30:36.928181 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:36.928162    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/kube-rbac-proxy-node/0.log"
Jul 09 18:30:36.965260 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:36.965233    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:30:36.991770 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:36.991744    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/northd/0.log"
Jul 09 18:30:37.021114 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:37.021091    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/nbdb/0.log"
Jul 09 18:30:37.047849 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:37.047822    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/sbdb/0.log"
Jul 09 18:30:37.093160 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:37.093132    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8jhz_08138fce-4009-438a-bf62-a17c4f40b74e/ovnkube-controller/0.log"
Jul 09 18:30:38.503726 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:38.503693    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fp6xc_1f662f2f-4128-4b2d-be2a-e3053a6f47e5/network-check-target-container/0.log"
Jul 09 18:30:39.536808 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:39.536782    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-vcgq4_d62dd174-274a-4a5a-99a3-069c5638ae31/iptables-alerter/0.log"
Jul 09 18:30:40.265025 ip-10-0-140-229 kubenswrapper[2561]: I0709 18:30:40.264997    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-swk86_249ac48c-521a-4576-a202-28091f9257cd/tuned/0.log"