Jul 07 16:45:31.686952 ip-10-0-142-63 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 16:45:31.686961 ip-10-0-142-63 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 16:45:31.686968 ip-10-0-142-63 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 16:45:31.687204 ip-10-0-142-63 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 16:45:41.918839 ip-10-0-142-63 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 16:45:41.918858 ip-10-0-142-63 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f770861f927842f2befab9d86734d416 --
Jul 07 16:47:39.873978 ip-10-0-142-63 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 16:47:40.345390 ip-10-0-142-63 systemd[1]: Started Kubernetes Kubelet.
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.268916    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273323    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273336    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273339    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273343    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273347    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273349    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273352    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:40.348533 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273355    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273358    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273361    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273363    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273366    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273368    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273371    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273374    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273377    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273385    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273388    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273390    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273394    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273396    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273399    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273401    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273404    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273406    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273409    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273412    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:40.350777 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273414    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273417    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273420    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273422    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273425    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273428    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273430    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273433    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273435    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273438    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273440    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273444    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273449    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273452    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273455    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273458    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273461    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273463    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273467    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:40.351682 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273470    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273472    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273475    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273477    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273480    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273483    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273486    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273488    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273492    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273496    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273499    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273502    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273506    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273509    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273512    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273515    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273517    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273520    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273522    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273525    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:40.352774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273527    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273532    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273535    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273537    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273540    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273543    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273546    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273548    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273551    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273554    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273556    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273560    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273563    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273566    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273568    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273571    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273574    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273576    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273578    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:40.361396 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273581    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273584    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273586    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273589    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273591    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273593    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273597    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273600    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273602    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273604    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273607    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273610    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273613    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.273615    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274328    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274335    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274338    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274341    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274343    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274347    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:40.363013 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274349    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274352    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274355    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274357    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274360    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274362    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274365    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274367    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274370    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274372    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274375    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274377    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274380    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274382    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274385    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274387    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274390    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274392    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274395    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274398    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:40.364030 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274400    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274403    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274405    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274408    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274411    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274414    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274416    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274418    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274421    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274425    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274429    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274433    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274437    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274440    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274443    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274446    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274448    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274451    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274454    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:40.366487 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274456    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274459    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274461    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274464    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274466    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274470    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274473    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274475    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274478    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274481    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274483    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274486    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274489    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274492    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274495    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274497    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274499    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274502    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274504    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274507    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:40.367552 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274509    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274512    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274514    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274517    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274520    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274522    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274525    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274528    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274530    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274533    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274535    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274538    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274540    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274543    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274545    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274547    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274550    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274552    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274555    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274558    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:40.368577 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274561    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274564    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274566    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274569    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274571    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274574    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274577    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274579    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274581    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274584    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274587    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274589    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274593    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274596    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274991    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274996    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.274999    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275002    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275005    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:40.372355 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275008    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275011    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275013    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275016    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275019    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275023    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275026    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275030    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275032    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275035    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275038    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275040    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275044    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275048    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275051    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275054    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275057    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275060    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275062    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:40.373234 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275066    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275068    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275071    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275073    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275076    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275078    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275082    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275085    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275088    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275091    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275093    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275096    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275099    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275102    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275105    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275107    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275110    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275112    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275116    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:42.219635 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275119    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275121    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275123    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275126    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275128    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275131    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275134    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275136    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275139    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275142    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275144    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275147    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275150    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275152    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275156    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275158    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275161    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275163    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275166    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275169    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:42.220877 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275171    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275174    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275176    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275178    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275181    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275183    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275186    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275188    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275191    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275193    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275197    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275202    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275205    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275207    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275210    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275212    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275214    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275217    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275219    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275222    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:42.223235 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275226    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275229    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275231    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275234    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275236    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275239    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275241    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275244    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275246    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275249    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275251    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275253    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275256    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275258    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275261    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275263    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275335    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275342    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275347    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275351    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275356    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 16:47:42.224003 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275359    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275363    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275369    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275372    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275375    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275378    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275381    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275384    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275387    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275389    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275392    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275396    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275399    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275403    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275406    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275409    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275412    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275416    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275419    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275422    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275425    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275428    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275431    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275434    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 16:47:42.224792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275437    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275440    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275443    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275447    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275450    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275453    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275456    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275458    2559 flags.go:64] FLAG: --enable-server="true"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275462    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275466    2559 flags.go:64] FLAG: --event-burst="100"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275470    2559 flags.go:64] FLAG: --event-qps="50"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275473    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275476    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275479    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275482    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275486    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275488    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275491    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275494    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275498    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275501    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275504    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275507    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275510    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275513    2559 flags.go:64] FLAG: --feature-gates=""
Jul 07 16:47:42.225571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275516    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275519    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275522    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275525    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275528    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275531    2559 flags.go:64] FLAG: --help="false"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275534    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-142-63.ec2.internal"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275537    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275539    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275542    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275545    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275549    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275552    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275554    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275557    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275560    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275566    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275571    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275574    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275577    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275579    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275582    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275585    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275588    2559 flags.go:64] FLAG: --lock-file=""
Jul 07 16:47:42.226380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275590    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275593    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275597    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275602    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275605    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275608    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275611    2559 flags.go:64] FLAG: --logging-format="text"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275613    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275617    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275619    2559 flags.go:64] FLAG: --manifest-url=""
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275622    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275626    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275629    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275637    2559 flags.go:64] FLAG: --max-pods="110"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275640    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275643    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275646    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275649    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275652    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275655    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275658    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275682    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275686    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275689    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 16:47:42.227219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275692    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275697    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275702    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275705    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275708    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275711    2559 flags.go:64] FLAG: --port="10250"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275714    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275716    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c82c3e3aa416c9f0"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275720    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275722    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275726    2559 flags.go:64] FLAG: --register-node="true"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275730    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275733    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275736    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275739    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275742    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275745    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275748    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275751    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275754    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275757    2559 flags.go:64] FLAG: --runonce="false"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275759    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275762    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275766    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275769    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 16:47:42.228017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275772    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275775    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275778    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275781    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275783    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275786    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275789    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275792    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275796    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275799    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275802    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275807    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275809    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275812    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275816    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275819    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275821    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275824    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275828    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275831    2559 flags.go:64] FLAG: --v="2"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275835    2559 flags.go:64] FLAG: --version="false"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275838    2559 flags.go:64] FLAG: --vmodule=""
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275856    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.275860    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275968    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:42.228803 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275972    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275975    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275978    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275981    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275984    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275987    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275989    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275992    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275995    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.275997    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276000    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276003    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276005    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276008    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276011    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276015    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276017    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276020    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276023    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276025    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:42.229586 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276027    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276030    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276032    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276035    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276037    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276041    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276043    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276046    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276048    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276051    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276053    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276056    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276059    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276062    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276064    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276067    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276070    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276074    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276077    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:42.230377 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276079    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276082    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276084    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276087    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276089    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276092    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276094    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276097    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276100    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276103    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276106    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276109    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276111    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276114    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276116    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276119    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276121    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276124    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276129    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276132    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:42.231199 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276135    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276138    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276141    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276144    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276146    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276149    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276152    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276155    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276157    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276160    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276162    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276165    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276167    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276170    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276172    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276174    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276177    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276179    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276182    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276184    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:42.231969 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276188    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276191    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276194    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276197    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276200    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276203    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276206    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276209    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276212    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276214    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276218    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276220    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276223    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276225    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276228    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276230    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276233    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276235    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:42.233965 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.276238    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.276913    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.284190    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.284207    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.284231    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.284238    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284333    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284340    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284345    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284351    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284357    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284361    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284366    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284371    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:42.234774 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284375    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284380    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284384    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284389    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284394    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284399    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284405    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284410    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284414    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284418    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284422    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284426    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284429    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284433    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284437    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284441    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284445    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284449    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284453    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:42.237185 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284458    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284462    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284466    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284471    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284475    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284479    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284484    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284490    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284493    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284498    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284502    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284506    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284510    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284514    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284518    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284523    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284527    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284532    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284536    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284540    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:42.238003 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284544    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284548    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284552    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284556    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284560    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284564    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284567    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284571    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284575    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284579    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284582    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284586    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284591    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284595    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284599    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284602    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284606    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284610    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284613    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284618    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:42.240006 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284622    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284626    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284629    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284633    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284637    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284641    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284645    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284649    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284653    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284657    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284674    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284683    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284686    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284690    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284694    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284698    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284702    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284706    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284710    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:42.240800 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284714    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284717    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284721    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284725    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284729    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284733    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284737    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284741    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284745    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284749    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284753    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284757    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284761    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.284767    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284926    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:47:42.241611 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284933    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284938    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284942    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284946    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284950    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284953    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284957    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284961    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284965    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284969    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284980    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284984    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284988    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284992    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.284996    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285000    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285004    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285009    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285015    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285019    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:47:42.242434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285023    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285028    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285032    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285036    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285039    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285043    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285047    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285051    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285055    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285059    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285063    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285067    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285071    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285074    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285078    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285082    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285085    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285089    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285092    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285096    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:47:42.243226 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285100    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285105    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285110    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285115    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285118    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285122    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285126    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285130    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285133    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285137    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285141    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285145    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285149    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285152    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285156    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285159    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285163    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285168    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285172    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285175    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:47:42.245229 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285179    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285183    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285187    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285191    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285195    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285199    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285202    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285206    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285210    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285215    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285218    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285223    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285227    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285232    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285237    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285243    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285248    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285253    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:47:42.246037 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285257    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285261    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285265    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285268    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285273    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285277    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285281    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285284    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285288    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285292    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285296    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285300    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285304    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285308    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285312    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285315    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285320    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285324    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285328    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:47:42.246832 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:40.285332    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.285339    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.285460    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.288913    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.290469    2559 server.go:1019] "Starting client certificate rotation"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.290568    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.290608    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.293479    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.300761    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.308433    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.311119    2559 log.go:25] "Validated CRI v1 image API"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.312697    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 16:47:42.344528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.315287    2559 fs.go:135] Filesystem UUIDs: map[78c91ca4-4084-4bf5-96e8-dd6721556f3d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f81c046a-ad7b-4f81-8268-18c541f111b2:/dev/nvme0n1p4]
Jul 07 16:47:42.346473 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.315305    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 16:47:42.346473 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.319054    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.321273    2559 manager.go:216] Machine: {Timestamp:2026-07-07 16:47:40.319907634 +0000 UTC m=+0.330755803 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099098 MemoryCapacity:33164505088 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec284bf35ab7229c10052393d261c07c SystemUUID:ec284bf3-5ab7-229c-1005-2393d261c07c BootID:f770861f-9278-42f2-befa-b9d86734d416 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582254592 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3e:d8:bf:24:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3e:d8:bf:24:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:8f:00:fb:29:9c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164505088 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.321374    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.321476    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.325358    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.325386    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-63.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.326334    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.326344    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.326351    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.327849    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.329054    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.331999    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.332103    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.332143    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.332159    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.332176    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.332184    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.332196    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 16:47:42.348099 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.332551    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.332578    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.336339    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.338093    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.338129    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339702    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339715    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339721    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339728    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339736    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339741    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339749    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339754    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339761    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339766    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339774    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.339779    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.341205    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.341222    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.343954    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-63.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:40.344004    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:40.344076    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.344458    2559 server.go:1268] "Started kubelet"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.344515    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.344616    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.344689    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.345765    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.346982    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.349096    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2ntvq"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:40.349587    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:40.349264    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-63.ec2.internal.18c00fd518095c52  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-63.ec2.internal,UID:ip-10-0-142-63.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-63.ec2.internal,},FirstTimestamp:2026-07-07 16:47:40.344433746 +0000 UTC m=+0.355281915,LastTimestamp:2026-07-07 16:47:40.344433746 +0000 UTC m=+0.355281915,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-63.ec2.internal,}"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.353042    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.353067    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:40.353270    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.353815    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.353823    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.353840    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.353911    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.353921    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.354019    2559 factory.go:55] Registering systemd factory
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.354034    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 07 16:47:42.348899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.354229    2559 factory.go:153] Registering CRI-O factory
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.354242    2559 factory.go:223] Registration of the crio container factory successfully
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.354290    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.354313    2559 factory.go:103] Registering Raw factory
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.354329    2559 manager.go:1196] Started watching for new ooms in manager
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.354758    2559 manager.go:349] Starting recovery of all containers
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:40.356315    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:40.356351    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.357549    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2ntvq"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.367101    2559 manager.go:354] Recovery completed
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.372405    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.374129    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.374156    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.374168    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.374693    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.374705    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.374706    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.374751    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.375317    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.375352    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.375366    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.375828    2559 policy_none.go:49] "None policy: Start"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.375840    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.375850    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:40.376948    2559 policy_none.go:47] "Start"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:42.200836    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:42.301095    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:42.350944 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:42.309361    2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-63.ec2.internal\" not found" node="ip-10-0-142-63.ec2.internal"
Jul 07 16:47:42.465638 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:42.401539    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:42.582873 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:42.502446    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:42.604493 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:42.603073    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:42.707702 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:42.703463    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:42.830317 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:42.803857    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:42.955592 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:42.904330    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.064157 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.004589    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.105015 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.104997    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.163601 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.163542    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 16:42:42 +0000 UTC" deadline="2028-03-19 03:56:02.63374367 +0000 UTC"
Jul 07 16:47:43.163687 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.163600    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14891h8m19.470147606s"
Jul 07 16:47:43.163687 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.163619    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 16:47:43.178627 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.178611    2559 manager.go:337] "Starting Device Plugin manager"
Jul 07 16:47:43.178730 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.178660    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 16:47:43.178730 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.178674    2559 server.go:87] "Starting device plugin registration server"
Jul 07 16:47:43.178951 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.178938    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 16:47:43.179022 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.178953    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 16:47:43.179067 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.179050    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 16:47:43.179158 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.179144    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 16:47:43.179158 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.179157    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 16:47:43.179754 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.179735    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 16:47:43.179830 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.179789    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.180459 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.180443    2559 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-63.ec2.internal" not found
Jul 07 16:47:43.197248 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.197230    2559 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-63.ec2.internal" not found
Jul 07 16:47:43.254175 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.254156    2559 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-63.ec2.internal" not found
Jul 07 16:47:43.279429 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.279408    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.280301 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.280286    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.280387 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.280310    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.280387 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.280320    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.280387 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.280344    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.283604 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.283586    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:47:43.288613 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.288598    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.288672 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.288618    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-63.ec2.internal\": node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.311311 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.311290    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.329137 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.329088    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 16:47:43.330157 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.330142    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 16:47:43.330234 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.330174    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 16:47:43.330234 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.330190    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 16:47:43.330234 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.330223    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 16:47:43.332478 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.332463    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:47:43.411801 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.411780    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.430954 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.430926    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"]
Jul 07 16:47:43.431073 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.431009    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.431844 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.431831    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.431909 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.431859    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.431909 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.431873    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.433321 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.433309    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.433448 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.433421    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.433520 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.433452    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.433969 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.433955    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.434050 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.433957    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.434050 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.434026    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.434050 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.434037    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.434050 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.433996    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.434175 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.434064    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.435612 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.435595    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.435675 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.435630    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:47:43.436266 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.436250    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:47:43.436347 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.436270    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:47:43.436347 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.436278    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:47:43.468674 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.468659    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-63.ec2.internal\" not found" node="ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.473147 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.473132    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-63.ec2.internal\" not found" node="ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.508531 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.508509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/512194b8ab62074e01ca53d7650baf23-config\") pod \"kube-apiserver-proxy-ip-10-0-142-63.ec2.internal\" (UID: \"512194b8ab62074e01ca53d7650baf23\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.508608 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.508551    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07d86b1c3b23c408d59f2c703c572ead-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal\" (UID: \"07d86b1c3b23c408d59f2c703c572ead\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.508608 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.508567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07d86b1c3b23c408d59f2c703c572ead-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal\" (UID: \"07d86b1c3b23c408d59f2c703c572ead\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.512494 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.512479    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.544825 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.544807    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:47:43.609373 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.609304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07d86b1c3b23c408d59f2c703c572ead-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal\" (UID: \"07d86b1c3b23c408d59f2c703c572ead\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.609373 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.609331    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/512194b8ab62074e01ca53d7650baf23-config\") pod \"kube-apiserver-proxy-ip-10-0-142-63.ec2.internal\" (UID: \"512194b8ab62074e01ca53d7650baf23\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.609373 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.609348    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07d86b1c3b23c408d59f2c703c572ead-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal\" (UID: \"07d86b1c3b23c408d59f2c703c572ead\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.609551 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.609375    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07d86b1c3b23c408d59f2c703c572ead-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal\" (UID: \"07d86b1c3b23c408d59f2c703c572ead\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.609551 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.609406    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07d86b1c3b23c408d59f2c703c572ead-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal\" (UID: \"07d86b1c3b23c408d59f2c703c572ead\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.609551 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.609424    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/512194b8ab62074e01ca53d7650baf23-config\") pod \"kube-apiserver-proxy-ip-10-0-142-63.ec2.internal\" (UID: \"512194b8ab62074e01ca53d7650baf23\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.613410 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.613392    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.713898 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.713860    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.772073 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.772040    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.777024 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:43.777011    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:43.814536 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.814507    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:43.915080 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:43.915000    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:44.015452 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:44.015427    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:44.116183 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:44.116152    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-63.ec2.internal\" not found"
Jul 07 16:47:44.166789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:44.166771    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:47:44.201773 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:44.201749    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:44.218401 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:44.218378    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:47:44.219434 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:44.219413    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal"
Jul 07 16:47:44.236655 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:44.236639    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:47:44.266466 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:44.266434    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d86b1c3b23c408d59f2c703c572ead.slice/crio-2e1b227b0464788034603e4c46856f18279e214ca0336f7094a3d55c657f236b WatchSource:0}: Error finding container 2e1b227b0464788034603e4c46856f18279e214ca0336f7094a3d55c657f236b: Status 404 returned error can't find the container with id 2e1b227b0464788034603e4c46856f18279e214ca0336f7094a3d55c657f236b
Jul 07 16:47:44.266888 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:44.266869    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512194b8ab62074e01ca53d7650baf23.slice/crio-7b71ee83cecad7589cf838473aae4f45c13522bd13204ebca80e28fc7a118945 WatchSource:0}: Error finding container 7b71ee83cecad7589cf838473aae4f45c13522bd13204ebca80e28fc7a118945: Status 404 returned error can't find the container with id 7b71ee83cecad7589cf838473aae4f45c13522bd13204ebca80e28fc7a118945
Jul 07 16:47:44.271432 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:44.271419    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:47:44.332936 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:44.332895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal" event={"ID":"07d86b1c3b23c408d59f2c703c572ead","Type":"ContainerStarted","Data":"2e1b227b0464788034603e4c46856f18279e214ca0336f7094a3d55c657f236b"}
Jul 07 16:47:44.333892 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:44.333868    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal" event={"ID":"512194b8ab62074e01ca53d7650baf23","Type":"ContainerStarted","Data":"7b71ee83cecad7589cf838473aae4f45c13522bd13204ebca80e28fc7a118945"}
Jul 07 16:47:45.081303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.081032    2559 apiserver.go:52] "Watching apiserver"
Jul 07 16:47:45.087784 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.087762    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 16:47:45.090285 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.090256    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-mtp24","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2","openshift-image-registry/node-ca-ftff6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal","openshift-multus/multus-64nzh","openshift-multus/multus-additional-cni-plugins-gg2f8","openshift-network-diagnostics/network-check-target-pjtgp","openshift-network-operator/iptables-alerter-m5zt9","kube-system/global-pull-secret-syncer-c57bv","kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal","openshift-cluster-node-tuning-operator/tuned-kz8s2","openshift-dns/node-resolver-tvtzh","openshift-multus/network-metrics-daemon-vp8v6","openshift-ovn-kubernetes/ovnkube-node-vkhn9"]
Jul 07 16:47:45.092025 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.092002    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:45.092131 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.092088    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:47:45.093402 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.093382    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.094952 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.094935    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.095649 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.095629    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.095904 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.095885    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jmj9k\""
Jul 07 16:47:45.096004 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.095888    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.096004 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.095888    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 16:47:45.096739 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.096635    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:47:45.097738 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.097158    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 16:47:45.097738 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.097233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.097738 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.097373    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pktdb\""
Jul 07 16:47:45.097738 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.097424    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.098604 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.098241    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.098840 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.098781    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 16:47:45.098930 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.098873    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m48fk\""
Jul 07 16:47:45.099817 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.099798    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.100704 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.100683    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 16:47:45.100849 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.100831    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-795k2\""
Jul 07 16:47:45.100919 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.100886    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.101036 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.101015    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.101609 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.101225    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 16:47:45.101609 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.101291    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:45.101609 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.101349    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:47:45.101609 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.101453    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 16:47:45.101890 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.101849    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 16:47:45.102126 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.102102    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpnr5\""
Jul 07 16:47:45.102486 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.102451    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 16:47:45.104773 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.104347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.106773 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.106746    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.106868 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.106806    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:47:45.107503 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.107002    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.107503 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.107121    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.107503 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.107215    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 16:47:45.107503 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.107380    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ql8q6\""
Jul 07 16:47:45.111517 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.111500    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.113384 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.113366    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 16:47:45.113821 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.113803    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7lhcz\""
Jul 07 16:47:45.114036 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.114020    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 16:47:45.115734 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.114927    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.115734 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.115208    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.115734 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.115245    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.115734 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.115457    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 16:47:45.117140 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.116989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-cnibin\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.117140 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117044    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/709bde3f-4f89-4234-8ec7-c96fcb8333f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.117140 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszfr\" (UniqueName: \"kubernetes.io/projected/dfa8416e-b75a-41d8-9d23-9d3aee893e52-kube-api-access-dszfr\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.117140 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:45.117140 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117144    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-registration-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.117939 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117175    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-etc-selinux\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.117939 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9787f13-a9f7-4e87-9051-2c50eeb65db0-cni-binary-copy\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.117939 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117233    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-run-k8s-cni-cncf-io\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.117939 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117268    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-daemon-config\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.117939 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117292    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-run-multus-certs\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.117939 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bkd\" (UniqueName: \"kubernetes.io/projected/4a1625b5-d332-4801-8459-07c873619a65-kube-api-access-v4bkd\") pod \"node-ca-ftff6\" (UID: \"4a1625b5-d332-4801-8459-07c873619a65\") " pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.117939 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-os-release\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.117939 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da05c267-ecba-4f27-883a-f5f9792142a0-iptables-alerter-script\") pod \"iptables-alerter-m5zt9\" (UID: \"da05c267-ecba-4f27-883a-f5f9792142a0\") " pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.117939 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.117897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.118371 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118077    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-system-cni-dir\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.118371 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-socket-dir-parent\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.118371 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a1625b5-d332-4801-8459-07c873619a65-serviceca\") pod \"node-ca-ftff6\" (UID: \"4a1625b5-d332-4801-8459-07c873619a65\") " pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.118371 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/374eb757-0763-44cb-864e-3145f83715a4-konnectivity-ca\") pod \"konnectivity-agent-mtp24\" (UID: \"374eb757-0763-44cb-864e-3145f83715a4\") " pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:47:45.118553 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:45.118553 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118459    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/374eb757-0763-44cb-864e-3145f83715a4-agent-certs\") pod \"konnectivity-agent-mtp24\" (UID: \"374eb757-0763-44cb-864e-3145f83715a4\") " pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:47:45.118655 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-system-cni-dir\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.118708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da05c267-ecba-4f27-883a-f5f9792142a0-host-slash\") pod \"iptables-alerter-m5zt9\" (UID: \"da05c267-ecba-4f27-883a-f5f9792142a0\") " pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.118755 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118724    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-device-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118864    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-sys-fs\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-run-netns\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.118925    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-hostroot\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-conf-dir\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-os-release\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119235    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-var-lib-cni-multus\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119263    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-var-lib-kubelet\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdxqw\" (UniqueName: \"kubernetes.io/projected/da05c267-ecba-4f27-883a-f5f9792142a0-kube-api-access-pdxqw\") pod \"iptables-alerter-m5zt9\" (UID: \"da05c267-ecba-4f27-883a-f5f9792142a0\") " pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-var-lib-cni-bin\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqlk\" (UniqueName: \"kubernetes.io/projected/b9787f13-a9f7-4e87-9051-2c50eeb65db0-kube-api-access-bjqlk\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dvt\" (UniqueName: \"kubernetes.io/projected/fb7c2897-e46e-456b-b3d2-7f419224c6b3-kube-api-access-n2dvt\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a1625b5-d332-4801-8459-07c873619a65-host\") pod \"node-ca-ftff6\" (UID: \"4a1625b5-d332-4801-8459-07c873619a65\") " pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/709bde3f-4f89-4234-8ec7-c96fcb8333f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119873    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5gdtl\""
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.119904    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/709bde3f-4f89-4234-8ec7-c96fcb8333f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.121443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.120227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxll\" (UniqueName: \"kubernetes.io/projected/709bde3f-4f89-4234-8ec7-c96fcb8333f5-kube-api-access-scxll\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.122455 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.120264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-socket-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.122455 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.120290    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-cni-dir\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.122455 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.120426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-cnibin\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.122455 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.120462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-etc-kubernetes\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.122455 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.120468    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.122455 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.121123    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.122455 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.121517    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 16:47:45.127773 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.126298    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 16:47:45.127773 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.126890    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5khqq\""
Jul 07 16:47:45.127773 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.127165    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.127773 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.127479    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 16:47:45.165398 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.165329    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 16:47:45.188111 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.188089    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:47:45.201916 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.201895    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 16:47:45.220991 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.220955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/709bde3f-4f89-4234-8ec7-c96fcb8333f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.221090 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxll\" (UniqueName: \"kubernetes.io/projected/709bde3f-4f89-4234-8ec7-c96fcb8333f5-kube-api-access-scxll\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.221172 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221153    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-socket-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.221250 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-cni-dir\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.221250 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-cnibin\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.221250 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221234    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.221393 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-run-openvswitch\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.221393 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-sysctl-d\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.221393 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221309    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-cnibin\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.221393 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221319    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-socket-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.221393 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszfr\" (UniqueName: \"kubernetes.io/projected/dfa8416e-b75a-41d8-9d23-9d3aee893e52-kube-api-access-dszfr\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.221393 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-var-lib-openvswitch\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.221393 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221386    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221412    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p469w\" (UniqueName: \"kubernetes.io/projected/67a4be0d-4845-43ec-a5d3-2952c9a6d01f-kube-api-access-p469w\") pod \"node-resolver-tvtzh\" (UID: \"67a4be0d-4845-43ec-a5d3-2952c9a6d01f\") " pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-registration-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9787f13-a9f7-4e87-9051-2c50eeb65db0-cni-binary-copy\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221462    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-cni-dir\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-run-k8s-cni-cncf-io\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-daemon-config\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-run-netns\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221565    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-run-k8s-cni-cncf-io\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221591    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/709bde3f-4f89-4234-8ec7-c96fcb8333f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221641    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-registration-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-tmp\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-os-release\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221739    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da05c267-ecba-4f27-883a-f5f9792142a0-iptables-alerter-script\") pod \"iptables-alerter-m5zt9\" (UID: \"da05c267-ecba-4f27-883a-f5f9792142a0\") " pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-socket-dir-parent\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.221789 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221792    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-os-release\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/374eb757-0763-44cb-864e-3145f83715a4-konnectivity-ca\") pod \"konnectivity-agent-mtp24\" (UID: \"374eb757-0763-44cb-864e-3145f83715a4\") " pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-sysctl-conf\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221849    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67a4be0d-4845-43ec-a5d3-2952c9a6d01f-hosts-file\") pod \"node-resolver-tvtzh\" (UID: \"67a4be0d-4845-43ec-a5d3-2952c9a6d01f\") " pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-run-systemd\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-systemd\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-run\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.221968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-host\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a4be0d-4845-43ec-a5d3-2952c9a6d01f-tmp-dir\") pod \"node-resolver-tvtzh\" (UID: \"67a4be0d-4845-43ec-a5d3-2952c9a6d01f\") " pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-sys-fs\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222050    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-socket-dir-parent\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-run-netns\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-hostroot\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222057    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9787f13-a9f7-4e87-9051-2c50eeb65db0-cni-binary-copy\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222143    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-hostroot\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222158    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-sys-fs\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.222581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222194    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-run-netns\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-node-log\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.222226    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-os-release\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222268    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-var-lib-cni-multus\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.222289    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs podName:fb7c2897-e46e-456b-b3d2-7f419224c6b3 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:45.722270488 +0000 UTC m=+3.985807113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs") pod "network-metrics-daemon-vp8v6" (UID: "fb7c2897-e46e-456b-b3d2-7f419224c6b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222299    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-var-lib-cni-multus\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-var-lib-kubelet\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d273870c-4ed6-44f4-be7c-669c1c013396-kubelet-config\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222352    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-os-release\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-sysconfig\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222385    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da05c267-ecba-4f27-883a-f5f9792142a0-iptables-alerter-script\") pod \"iptables-alerter-m5zt9\" (UID: \"da05c267-ecba-4f27-883a-f5f9792142a0\") " pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222400    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-var-lib-kubelet\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222396    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxqw\" (UniqueName: \"kubernetes.io/projected/da05c267-ecba-4f27-883a-f5f9792142a0-kube-api-access-pdxqw\") pod \"iptables-alerter-m5zt9\" (UID: \"da05c267-ecba-4f27-883a-f5f9792142a0\") " pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222452    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqlk\" (UniqueName: \"kubernetes.io/projected/b9787f13-a9f7-4e87-9051-2c50eeb65db0-kube-api-access-bjqlk\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dvt\" (UniqueName: \"kubernetes.io/projected/fb7c2897-e46e-456b-b3d2-7f419224c6b3-kube-api-access-n2dvt\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a1625b5-d332-4801-8459-07c873619a65-host\") pod \"node-ca-ftff6\" (UID: \"4a1625b5-d332-4801-8459-07c873619a65\") " pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.223399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222503    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/374eb757-0763-44cb-864e-3145f83715a4-konnectivity-ca\") pod \"konnectivity-agent-mtp24\" (UID: \"374eb757-0763-44cb-864e-3145f83715a4\") " pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222516    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/282109b7-c9be-4d64-a867-4281aa2b16b9-ovnkube-config\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222562    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a1625b5-d332-4801-8459-07c873619a65-host\") pod \"node-ca-ftff6\" (UID: \"4a1625b5-d332-4801-8459-07c873619a65\") " pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/282109b7-c9be-4d64-a867-4281aa2b16b9-ovnkube-script-lib\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-tuned\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222765    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-daemon-config\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/709bde3f-4f89-4234-8ec7-c96fcb8333f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-etc-kubernetes\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222836    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-cni-netd\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222940    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-etc-kubernetes\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.222989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-cnibin\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/709bde3f-4f89-4234-8ec7-c96fcb8333f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223085    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-cnibin\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-run-ovn\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/282109b7-c9be-4d64-a867-4281aa2b16b9-env-overrides\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223285    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/709bde3f-4f89-4234-8ec7-c96fcb8333f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.224233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-etc-selinux\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-run-multus-certs\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223353    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-etc-selinux\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bkd\" (UniqueName: \"kubernetes.io/projected/4a1625b5-d332-4801-8459-07c873619a65-kube-api-access-v4bkd\") pod \"node-ca-ftff6\" (UID: \"4a1625b5-d332-4801-8459-07c873619a65\") " pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223386    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-slash\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-cni-bin\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223434    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-system-cni-dir\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223436    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-run-multus-certs\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223458    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a1625b5-d332-4801-8459-07c873619a65-serviceca\") pod \"node-ca-ftff6\" (UID: \"4a1625b5-d332-4801-8459-07c873619a65\") " pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d273870c-4ed6-44f4-be7c-669c1c013396-dbus\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223503    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-system-cni-dir\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223503    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-etc-openvswitch\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223513    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/709bde3f-4f89-4234-8ec7-c96fcb8333f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-sys\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223555    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-var-lib-kubelet\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/374eb757-0763-44cb-864e-3145f83715a4-agent-certs\") pod \"konnectivity-agent-mtp24\" (UID: \"374eb757-0763-44cb-864e-3145f83715a4\") " pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/282109b7-c9be-4d64-a867-4281aa2b16b9-ovn-node-metrics-cert\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.224907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223615    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-modprobe-d\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-lib-modules\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-system-cni-dir\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223707    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-system-cni-dir\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da05c267-ecba-4f27-883a-f5f9792142a0-host-slash\") pod \"iptables-alerter-m5zt9\" (UID: \"da05c267-ecba-4f27-883a-f5f9792142a0\") " pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223767    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-device-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-conf-dir\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223802    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da05c267-ecba-4f27-883a-f5f9792142a0-host-slash\") pod \"iptables-alerter-m5zt9\" (UID: \"da05c267-ecba-4f27-883a-f5f9792142a0\") " pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-systemd-units\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-log-socket\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223848    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8416e-b75a-41d8-9d23-9d3aee893e52-device-dir\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223865    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jv5n\" (UniqueName: \"kubernetes.io/projected/282109b7-c9be-4d64-a867-4281aa2b16b9-kube-api-access-5jv5n\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223876    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223894    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a1625b5-d332-4801-8459-07c873619a65-serviceca\") pod \"node-ca-ftff6\" (UID: \"4a1625b5-d332-4801-8459-07c873619a65\") " pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223901    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-kubernetes\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-var-lib-cni-bin\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.225708 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.224018    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-kubelet\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.226460 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.224041    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.226460 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.224066    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mm4v\" (UniqueName: \"kubernetes.io/projected/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-kube-api-access-8mm4v\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.226460 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.224076    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-host-var-lib-cni-bin\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.226460 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.223895    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9787f13-a9f7-4e87-9051-2c50eeb65db0-multus-conf-dir\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.226460 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.224122    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/709bde3f-4f89-4234-8ec7-c96fcb8333f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.227724 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.227693    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bfd2q"
Jul 07 16:47:45.228036 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.228010    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/374eb757-0763-44cb-864e-3145f83715a4-agent-certs\") pod \"konnectivity-agent-mtp24\" (UID: \"374eb757-0763-44cb-864e-3145f83715a4\") " pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:47:45.237788 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.237761    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxll\" (UniqueName: \"kubernetes.io/projected/709bde3f-4f89-4234-8ec7-c96fcb8333f5-kube-api-access-scxll\") pod \"multus-additional-cni-plugins-gg2f8\" (UID: \"709bde3f-4f89-4234-8ec7-c96fcb8333f5\") " pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.238403 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.238378    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bfd2q"
Jul 07 16:47:45.238498 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.238420    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxqw\" (UniqueName: \"kubernetes.io/projected/da05c267-ecba-4f27-883a-f5f9792142a0-kube-api-access-pdxqw\") pod \"iptables-alerter-m5zt9\" (UID: \"da05c267-ecba-4f27-883a-f5f9792142a0\") " pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.245505 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.245484    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:45.245599 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.245509    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:45.245599 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.245522    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5kwd for pod openshift-network-diagnostics/network-check-target-pjtgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:45.245599 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.245576    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd podName:077f2741-19ab-4827-ace5-c11203efc785 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:45.745559277 +0000 UTC m=+4.009095904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k5kwd" (UniqueName: "kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd") pod "network-check-target-pjtgp" (UID: "077f2741-19ab-4827-ace5-c11203efc785") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:45.248044 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.248022    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dvt\" (UniqueName: \"kubernetes.io/projected/fb7c2897-e46e-456b-b3d2-7f419224c6b3-kube-api-access-n2dvt\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:45.248225 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.248207    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqlk\" (UniqueName: \"kubernetes.io/projected/b9787f13-a9f7-4e87-9051-2c50eeb65db0-kube-api-access-bjqlk\") pod \"multus-64nzh\" (UID: \"b9787f13-a9f7-4e87-9051-2c50eeb65db0\") " pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.249096 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.249052    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszfr\" (UniqueName: \"kubernetes.io/projected/dfa8416e-b75a-41d8-9d23-9d3aee893e52-kube-api-access-dszfr\") pod \"aws-ebs-csi-driver-node-fmcd2\" (UID: \"dfa8416e-b75a-41d8-9d23-9d3aee893e52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.249633 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.249611    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bkd\" (UniqueName: \"kubernetes.io/projected/4a1625b5-d332-4801-8459-07c873619a65-kube-api-access-v4bkd\") pod \"node-ca-ftff6\" (UID: \"4a1625b5-d332-4801-8459-07c873619a65\") " pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.324602 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-sysctl-conf\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.324602 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67a4be0d-4845-43ec-a5d3-2952c9a6d01f-hosts-file\") pod \"node-resolver-tvtzh\" (UID: \"67a4be0d-4845-43ec-a5d3-2952c9a6d01f\") " pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-run-systemd\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-systemd\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-run\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-host\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a4be0d-4845-43ec-a5d3-2952c9a6d01f-tmp-dir\") pod \"node-resolver-tvtzh\" (UID: \"67a4be0d-4845-43ec-a5d3-2952c9a6d01f\") " pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-node-log\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324747    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-run-systemd\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324732    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67a4be0d-4845-43ec-a5d3-2952c9a6d01f-hosts-file\") pod \"node-resolver-tvtzh\" (UID: \"67a4be0d-4845-43ec-a5d3-2952c9a6d01f\") " pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d273870c-4ed6-44f4-be7c-669c1c013396-kubelet-config\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324767    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-host\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324788    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-sysctl-conf\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.324813 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324817    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-run\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-sysconfig\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324825    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-systemd\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324831    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d273870c-4ed6-44f4-be7c-669c1c013396-kubelet-config\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324855    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-sysconfig\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324858    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/282109b7-c9be-4d64-a867-4281aa2b16b9-ovnkube-config\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324871    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-node-log\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/282109b7-c9be-4d64-a867-4281aa2b16b9-ovnkube-script-lib\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-tuned\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.324969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-cni-netd\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-run-ovn\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/282109b7-c9be-4d64-a867-4281aa2b16b9-env-overrides\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-slash\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-cni-bin\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325156    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a4be0d-4845-43ec-a5d3-2952c9a6d01f-tmp-dir\") pod \"node-resolver-tvtzh\" (UID: \"67a4be0d-4845-43ec-a5d3-2952c9a6d01f\") " pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d273870c-4ed6-44f4-be7c-669c1c013396-dbus\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-etc-openvswitch\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-sys\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.325415 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325329    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-var-lib-kubelet\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/282109b7-c9be-4d64-a867-4281aa2b16b9-ovn-node-metrics-cert\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325391    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-modprobe-d\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-lib-modules\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-systemd-units\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325442    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/282109b7-c9be-4d64-a867-4281aa2b16b9-ovnkube-config\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325472    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d273870c-4ed6-44f4-be7c-669c1c013396-dbus\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325497    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-slash\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325535    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-cni-bin\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325547    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/282109b7-c9be-4d64-a867-4281aa2b16b9-ovnkube-script-lib\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325553    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-cni-netd\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-log-socket\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325603    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-systemd-units\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jv5n\" (UniqueName: \"kubernetes.io/projected/282109b7-c9be-4d64-a867-4281aa2b16b9-kube-api-access-5jv5n\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325621    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/282109b7-c9be-4d64-a867-4281aa2b16b9-env-overrides\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-kubernetes\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325667    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-kubelet\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325676    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-lib-modules\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.326303 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325683    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-run-ovn\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325671    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-modprobe-d\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325721    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-log-socket\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325784    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-kubernetes\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mm4v\" (UniqueName: \"kubernetes.io/projected/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-kube-api-access-8mm4v\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325824    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-etc-openvswitch\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325831    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-var-lib-kubelet\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325845    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-sys\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325890    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-kubelet\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-run-openvswitch\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-sysctl-d\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.325968    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-var-lib-openvswitch\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p469w\" (UniqueName: \"kubernetes.io/projected/67a4be0d-4845-43ec-a5d3-2952c9a6d01f-kube-api-access-p469w\") pod \"node-resolver-tvtzh\" (UID: \"67a4be0d-4845-43ec-a5d3-2952c9a6d01f\") " pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-run-netns\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.327179 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.326102    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:45.328041 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-tmp\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.328041 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326165    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-sysctl-d\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.328041 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.326183    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret podName:d273870c-4ed6-44f4-be7c-669c1c013396 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:45.826159201 +0000 UTC m=+4.089695814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret") pod "global-pull-secret-syncer-c57bv" (UID: "d273870c-4ed6-44f4-be7c-669c1c013396") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:45.328041 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326216    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-run-openvswitch\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.328041 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326216    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-var-lib-openvswitch\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.328041 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326222    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-run-netns\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.328041 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.326287    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/282109b7-c9be-4d64-a867-4281aa2b16b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.328041 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.327358    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-etc-tuned\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.328305 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.328070    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/282109b7-c9be-4d64-a867-4281aa2b16b9-ovn-node-metrics-cert\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.328305 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.328152    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-tmp\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.336809 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.336739    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jv5n\" (UniqueName: \"kubernetes.io/projected/282109b7-c9be-4d64-a867-4281aa2b16b9-kube-api-access-5jv5n\") pod \"ovnkube-node-vkhn9\" (UID: \"282109b7-c9be-4d64-a867-4281aa2b16b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.337011 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.336993    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mm4v\" (UniqueName: \"kubernetes.io/projected/7b6ad925-dd67-4e6b-8bdc-98c37416d65f-kube-api-access-8mm4v\") pod \"tuned-kz8s2\" (UID: \"7b6ad925-dd67-4e6b-8bdc-98c37416d65f\") " pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.337257 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.337240    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p469w\" (UniqueName: \"kubernetes.io/projected/67a4be0d-4845-43ec-a5d3-2952c9a6d01f-kube-api-access-p469w\") pod \"node-resolver-tvtzh\" (UID: \"67a4be0d-4845-43ec-a5d3-2952c9a6d01f\") " pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.407757 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.407725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2"
Jul 07 16:47:45.433039 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.433013    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:47:45.437563 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.437543    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftff6"
Jul 07 16:47:45.442274 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.442242    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-64nzh"
Jul 07 16:47:45.451883 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.451867    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gg2f8"
Jul 07 16:47:45.467572 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.467550    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m5zt9"
Jul 07 16:47:45.473427 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.473407    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:47:45.480189 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.480173    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kz8s2"
Jul 07 16:47:45.487769 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.487752    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tvtzh"
Jul 07 16:47:45.729495 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.729419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:45.729656 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.729573    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:45.729656 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.729641    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs podName:fb7c2897-e46e-456b-b3d2-7f419224c6b3 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:46.729625881 +0000 UTC m=+4.993162502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs") pod "network-metrics-daemon-vp8v6" (UID: "fb7c2897-e46e-456b-b3d2-7f419224c6b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:45.830150 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.830117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:45.830309 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:45.830174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:45.830309 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.830263    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:45.830377 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.830325    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret podName:d273870c-4ed6-44f4-be7c-669c1c013396 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:46.830309773 +0000 UTC m=+5.093846387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret") pod "global-pull-secret-syncer-c57bv" (UID: "d273870c-4ed6-44f4-be7c-669c1c013396") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:45.830377 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.830271    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:45.830377 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.830368    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:45.830475 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.830382    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5kwd for pod openshift-network-diagnostics/network-check-target-pjtgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:45.830475 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:45.830428    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd podName:077f2741-19ab-4827-ace5-c11203efc785 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:46.830415008 +0000 UTC m=+5.093951619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5kwd" (UniqueName: "kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd") pod "network-check-target-pjtgp" (UID: "077f2741-19ab-4827-ace5-c11203efc785") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:46.241547 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.241498    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:42:45 +0000 UTC" deadline="2028-01-27 03:55:19.194658412 +0000 UTC"
Jul 07 16:47:46.241547 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.241538    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13643h7m32.953125427s"
Jul 07 16:47:46.291171 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:46.291133    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa8416e_b75a_41d8_9d23_9d3aee893e52.slice/crio-87f6d11432be5bc956b37bb2262e079142e8222b2e14e6b844747035006cec09 WatchSource:0}: Error finding container 87f6d11432be5bc956b37bb2262e079142e8222b2e14e6b844747035006cec09: Status 404 returned error can't find the container with id 87f6d11432be5bc956b37bb2262e079142e8222b2e14e6b844747035006cec09
Jul 07 16:47:46.298209 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:46.298187    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6ad925_dd67_4e6b_8bdc_98c37416d65f.slice/crio-9e537d9079faf0b6b188398e27f8b1be189bc4346d36c51372ad27b9d982d58f WatchSource:0}: Error finding container 9e537d9079faf0b6b188398e27f8b1be189bc4346d36c51372ad27b9d982d58f: Status 404 returned error can't find the container with id 9e537d9079faf0b6b188398e27f8b1be189bc4346d36c51372ad27b9d982d58f
Jul 07 16:47:46.302274 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:46.299753    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda05c267_ecba_4f27_883a_f5f9792142a0.slice/crio-0424a8d71773ee27d94607c2b33d2520c0ec128418f44703caa22125a7fcfe78 WatchSource:0}: Error finding container 0424a8d71773ee27d94607c2b33d2520c0ec128418f44703caa22125a7fcfe78: Status 404 returned error can't find the container with id 0424a8d71773ee27d94607c2b33d2520c0ec128418f44703caa22125a7fcfe78
Jul 07 16:47:46.305598 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:46.305573    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709bde3f_4f89_4234_8ec7_c96fcb8333f5.slice/crio-c675b9429bff34bf269ace4701895ae66e6b708274914d3edf77a5c26086ee41 WatchSource:0}: Error finding container c675b9429bff34bf269ace4701895ae66e6b708274914d3edf77a5c26086ee41: Status 404 returned error can't find the container with id c675b9429bff34bf269ace4701895ae66e6b708274914d3edf77a5c26086ee41
Jul 07 16:47:46.307107 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:46.306467    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9787f13_a9f7_4e87_9051_2c50eeb65db0.slice/crio-84230528531e3ef1ca627f73fc86a2ccbd478c1d912eedc694787c7df4d84815 WatchSource:0}: Error finding container 84230528531e3ef1ca627f73fc86a2ccbd478c1d912eedc694787c7df4d84815: Status 404 returned error can't find the container with id 84230528531e3ef1ca627f73fc86a2ccbd478c1d912eedc694787c7df4d84815
Jul 07 16:47:46.307549 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:47:46.307448    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1625b5_d332_4801_8459_07c873619a65.slice/crio-ef3c61ece142310fb5b5ea1fca6a6ec2c0cc804dbc58aecac222f0b36664b7bf WatchSource:0}: Error finding container ef3c61ece142310fb5b5ea1fca6a6ec2c0cc804dbc58aecac222f0b36664b7bf: Status 404 returned error can't find the container with id ef3c61ece142310fb5b5ea1fca6a6ec2c0cc804dbc58aecac222f0b36664b7bf
Jul 07 16:47:46.338512 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.338392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftff6" event={"ID":"4a1625b5-d332-4801-8459-07c873619a65","Type":"ContainerStarted","Data":"ef3c61ece142310fb5b5ea1fca6a6ec2c0cc804dbc58aecac222f0b36664b7bf"}
Jul 07 16:47:46.343904 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.343882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-64nzh" event={"ID":"b9787f13-a9f7-4e87-9051-2c50eeb65db0","Type":"ContainerStarted","Data":"84230528531e3ef1ca627f73fc86a2ccbd478c1d912eedc694787c7df4d84815"}
Jul 07 16:47:46.347892 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.347863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mtp24" event={"ID":"374eb757-0763-44cb-864e-3145f83715a4","Type":"ContainerStarted","Data":"b05345161bf6c47a023158de5fc0ea357c2f2772c40d817276b22d3451999a86"}
Jul 07 16:47:46.348827 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.348807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" event={"ID":"282109b7-c9be-4d64-a867-4281aa2b16b9","Type":"ContainerStarted","Data":"ea0bec4a6559c36a57935b85279cd1f654b62e441ee99030522b5d50112421dc"}
Jul 07 16:47:46.349854 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.349836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kz8s2" event={"ID":"7b6ad925-dd67-4e6b-8bdc-98c37416d65f","Type":"ContainerStarted","Data":"9e537d9079faf0b6b188398e27f8b1be189bc4346d36c51372ad27b9d982d58f"}
Jul 07 16:47:46.350874 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.350857    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tvtzh" event={"ID":"67a4be0d-4845-43ec-a5d3-2952c9a6d01f","Type":"ContainerStarted","Data":"5a1731704bf96fa5985b41fe490b18c2dbff6b475aa5d0fa250b89ee8f6697ce"}
Jul 07 16:47:46.351894 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.351873    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg2f8" event={"ID":"709bde3f-4f89-4234-8ec7-c96fcb8333f5","Type":"ContainerStarted","Data":"c675b9429bff34bf269ace4701895ae66e6b708274914d3edf77a5c26086ee41"}
Jul 07 16:47:46.352998 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.352964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m5zt9" event={"ID":"da05c267-ecba-4f27-883a-f5f9792142a0","Type":"ContainerStarted","Data":"0424a8d71773ee27d94607c2b33d2520c0ec128418f44703caa22125a7fcfe78"}
Jul 07 16:47:46.353968 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.353946    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2" event={"ID":"dfa8416e-b75a-41d8-9d23-9d3aee893e52","Type":"ContainerStarted","Data":"87f6d11432be5bc956b37bb2262e079142e8222b2e14e6b844747035006cec09"}
Jul 07 16:47:46.736733 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.736697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:46.736917 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:46.736900    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:46.737002 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:46.736965    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs podName:fb7c2897-e46e-456b-b3d2-7f419224c6b3 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:48.736947425 +0000 UTC m=+7.000484038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs") pod "network-metrics-daemon-vp8v6" (UID: "fb7c2897-e46e-456b-b3d2-7f419224c6b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:46.837851 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.837770    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:46.838016 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:46.837863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:46.838016 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:46.837914    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:46.838016 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:46.838003    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret podName:d273870c-4ed6-44f4-be7c-669c1c013396 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:48.837969568 +0000 UTC m=+7.101506184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret") pod "global-pull-secret-syncer-c57bv" (UID: "d273870c-4ed6-44f4-be7c-669c1c013396") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:46.838186 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:46.838050    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:46.838186 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:46.838073    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:46.838186 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:46.838087    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5kwd for pod openshift-network-diagnostics/network-check-target-pjtgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:46.838186 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:46.838136    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd podName:077f2741-19ab-4827-ace5-c11203efc785 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:48.838121731 +0000 UTC m=+7.101658359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5kwd" (UniqueName: "kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd") pod "network-check-target-pjtgp" (UID: "077f2741-19ab-4827-ace5-c11203efc785") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:47.242400 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:47.242274    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:42:45 +0000 UTC" deadline="2028-01-14 09:53:52.551304757 +0000 UTC"
Jul 07 16:47:47.242400 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:47.242315    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13337h6m5.308994571s"
Jul 07 16:47:47.333227 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:47.333200    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:47.333400 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:47.333318    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:47:47.333726 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:47.333705    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:47.333865 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:47.333822    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:47:47.333933 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:47.333890    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:47.334008 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:47.333987    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:47:47.380585 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:47.380545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal" event={"ID":"512194b8ab62074e01ca53d7650baf23","Type":"ContainerStarted","Data":"c06f8cc5677b5541cb8ab485806ca84694baf344d33a59d3c2c5da0f49d3d724"}
Jul 07 16:47:47.401945 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:47.400605    2559 generic.go:356] "Generic (PLEG): container finished" podID="07d86b1c3b23c408d59f2c703c572ead" containerID="7fb6abab55a77dfbe75745c44ce7f97774d357010482614802e4f45834585838" exitCode=0
Jul 07 16:47:47.401945 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:47.400655    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal" event={"ID":"07d86b1c3b23c408d59f2c703c572ead","Type":"ContainerDied","Data":"7fb6abab55a77dfbe75745c44ce7f97774d357010482614802e4f45834585838"}
Jul 07 16:47:47.426356 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:47.425134    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-63.ec2.internal" podStartSLOduration=3.425113735 podStartE2EDuration="3.425113735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:47.4055199 +0000 UTC m=+5.669056535" watchObservedRunningTime="2026-07-07 16:47:47.425113735 +0000 UTC m=+5.688650369"
Jul 07 16:47:48.431351 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:48.431274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal" event={"ID":"07d86b1c3b23c408d59f2c703c572ead","Type":"ContainerStarted","Data":"af61c157a68bf8d12c2dcc9dec9b415a8fbdfc1fa1624f9c521ba719c3efc6c6"}
Jul 07 16:47:48.461256 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:48.460994    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-63.ec2.internal" podStartSLOduration=4.460956657 podStartE2EDuration="4.460956657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:48.460489138 +0000 UTC m=+6.724025770" watchObservedRunningTime="2026-07-07 16:47:48.460956657 +0000 UTC m=+6.724493292"
Jul 07 16:47:48.756125 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:48.755522    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:48.756125 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:48.755702    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:48.756125 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:48.755769    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs podName:fb7c2897-e46e-456b-b3d2-7f419224c6b3 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:52.755749726 +0000 UTC m=+11.019286339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs") pod "network-metrics-daemon-vp8v6" (UID: "fb7c2897-e46e-456b-b3d2-7f419224c6b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:48.857061 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:48.856301    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:48.857061 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:48.856407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:48.857061 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:48.856579    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:48.857061 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:48.856601    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:48.857061 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:48.856615    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5kwd for pod openshift-network-diagnostics/network-check-target-pjtgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:48.857061 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:48.856674    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd podName:077f2741-19ab-4827-ace5-c11203efc785 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:52.856655077 +0000 UTC m=+11.120191693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5kwd" (UniqueName: "kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd") pod "network-check-target-pjtgp" (UID: "077f2741-19ab-4827-ace5-c11203efc785") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:48.857503 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:48.857080    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:48.857503 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:48.857132    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret podName:d273870c-4ed6-44f4-be7c-669c1c013396 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:52.857118155 +0000 UTC m=+11.120654780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret") pod "global-pull-secret-syncer-c57bv" (UID: "d273870c-4ed6-44f4-be7c-669c1c013396") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:49.330770 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:49.330737    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:49.330952 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:49.330867    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:47:49.331284 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:49.331251    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:49.331375 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:49.331354    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:47:49.331434 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:49.331421    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:49.331523 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:49.331488    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:47:51.330695 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:51.330662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:51.331197 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:51.330793    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:47:51.331284 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:51.331226    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:51.331345 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:51.331314    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:47:51.331402 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:51.331362    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:51.331488 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:51.331467    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:47:52.792929 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:52.792871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:52.793396 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:52.793098    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:52.793396 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:52.793159    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs podName:fb7c2897-e46e-456b-b3d2-7f419224c6b3 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:00.793141906 +0000 UTC m=+19.056678518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs") pod "network-metrics-daemon-vp8v6" (UID: "fb7c2897-e46e-456b-b3d2-7f419224c6b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:52.893624 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:52.893576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:52.893797 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:52.893651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:52.893797 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:52.893771    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:52.893902 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:52.893833    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret podName:d273870c-4ed6-44f4-be7c-669c1c013396 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:00.893813914 +0000 UTC m=+19.157350548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret") pod "global-pull-secret-syncer-c57bv" (UID: "d273870c-4ed6-44f4-be7c-669c1c013396") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:52.893990 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:52.893937    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:52.893990 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:52.893954    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:52.893990 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:52.893967    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5kwd for pod openshift-network-diagnostics/network-check-target-pjtgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:52.894132 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:52.894035    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd podName:077f2741-19ab-4827-ace5-c11203efc785 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:00.894020439 +0000 UTC m=+19.157557061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5kwd" (UniqueName: "kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd") pod "network-check-target-pjtgp" (UID: "077f2741-19ab-4827-ace5-c11203efc785") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:53.332991 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:53.332241    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:53.332991 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:53.332365    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:47:53.332991 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:53.332734    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:53.332991 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:53.332814    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:47:53.332991 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:53.332866    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:53.332991 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:53.332936    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:47:55.330469 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:55.330385    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:55.330989 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:55.330500    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:47:55.330989 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:55.330388    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:55.330989 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:55.330532    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:55.330989 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:55.330606    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:47:55.330989 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:55.330700    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:47:57.331027 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:57.330992    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:57.331477 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:57.330992    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:57.331477 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:57.331119    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:47:57.331477 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:57.331127    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:57.331477 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:57.331232    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:47:57.331477 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:57.331339    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:47:59.331340 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:59.331304    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:47:59.331340 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:59.331335    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:47:59.331812 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:47:59.331304    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:47:59.331812 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:59.331436    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:47:59.331812 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:59.331519    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:47:59.331812 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:47:59.331620    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:00.856643 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:00.856601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:00.857143 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:00.856742    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:48:00.857143 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:00.856809    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs podName:fb7c2897-e46e-456b-b3d2-7f419224c6b3 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:16.856791993 +0000 UTC m=+35.120328608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs") pod "network-metrics-daemon-vp8v6" (UID: "fb7c2897-e46e-456b-b3d2-7f419224c6b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:48:00.957270 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:00.957236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:00.957446 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:00.957309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:00.957446 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:00.957413    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:48:00.957446 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:00.957429    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:48:00.957446 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:00.957444    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:48:00.957634 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:00.957457    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5kwd for pod openshift-network-diagnostics/network-check-target-pjtgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:48:00.957634 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:00.957493    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret podName:d273870c-4ed6-44f4-be7c-669c1c013396 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:16.957471478 +0000 UTC m=+35.221008092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret") pod "global-pull-secret-syncer-c57bv" (UID: "d273870c-4ed6-44f4-be7c-669c1c013396") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:48:00.957634 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:00.957513    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd podName:077f2741-19ab-4827-ace5-c11203efc785 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:16.957503256 +0000 UTC m=+35.221039879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5kwd" (UniqueName: "kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd") pod "network-check-target-pjtgp" (UID: "077f2741-19ab-4827-ace5-c11203efc785") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:48:01.331380 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:01.331351    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:01.331549 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:01.331386    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:01.331549 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:01.331354    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:01.331549 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:01.331458    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:01.331549 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:01.331517    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:01.331728 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:01.331609    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:03.331314 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:03.331273    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:03.331776 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:03.331381    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:03.331776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:03.331454    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:03.331776 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:03.331543    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:03.331776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:03.331557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:03.331776 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:03.331679    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:05.332189 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.331531    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:05.332189 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:05.331863    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:05.332189 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.331943    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:05.332189 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:05.332028    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:05.332189 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.332090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:05.332189 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:05.332164    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:05.467082 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.466885    2559 generic.go:356] "Generic (PLEG): container finished" podID="709bde3f-4f89-4234-8ec7-c96fcb8333f5" containerID="4cacb6545cf2a3946e3352ef99829967438f538346320827fd9f2608888d3e89" exitCode=0
Jul 07 16:48:05.467175 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.466967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg2f8" event={"ID":"709bde3f-4f89-4234-8ec7-c96fcb8333f5","Type":"ContainerDied","Data":"4cacb6545cf2a3946e3352ef99829967438f538346320827fd9f2608888d3e89"}
Jul 07 16:48:05.468744 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.468664    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2" event={"ID":"dfa8416e-b75a-41d8-9d23-9d3aee893e52","Type":"ContainerStarted","Data":"473ba858115b9e323b9257973f4e59fa34da08289163000da23961eb8679d07a"}
Jul 07 16:48:05.470152 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.470125    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftff6" event={"ID":"4a1625b5-d332-4801-8459-07c873619a65","Type":"ContainerStarted","Data":"e8bfbb08c26015dea718ab7ac553d5d128750612dbb65f902158286bd8e47eb5"}
Jul 07 16:48:05.471598 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.471573    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-64nzh" event={"ID":"b9787f13-a9f7-4e87-9051-2c50eeb65db0","Type":"ContainerStarted","Data":"bffe881db3866027bdd6611f56b5e6d6ba9f45ba28c0ecd2c753e8f71938fbc8"}
Jul 07 16:48:05.473212 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.473192    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mtp24" event={"ID":"374eb757-0763-44cb-864e-3145f83715a4","Type":"ContainerStarted","Data":"1ba6c64c5bb55799f8555aa05744274596f91d0d39ddfa3524fadb1aa9e7fc9f"}
Jul 07 16:48:05.475890 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.475575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" event={"ID":"282109b7-c9be-4d64-a867-4281aa2b16b9","Type":"ContainerStarted","Data":"fd31f35e2d4b7a9d455989dbaedbae7197d3b49aa6aea9efa31220385b992c6a"}
Jul 07 16:48:05.475890 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.475653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" event={"ID":"282109b7-c9be-4d64-a867-4281aa2b16b9","Type":"ContainerStarted","Data":"5cb69d8e9c37d82182ed705dc432c04aa19c990e0b11856d89118af455c997f0"}
Jul 07 16:48:05.477439 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.477414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kz8s2" event={"ID":"7b6ad925-dd67-4e6b-8bdc-98c37416d65f","Type":"ContainerStarted","Data":"9ff33bc34a3bb682a2baf5672c0e43afb7cd8611e91a35f894fee59786d40c30"}
Jul 07 16:48:05.478895 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.478874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tvtzh" event={"ID":"67a4be0d-4845-43ec-a5d3-2952c9a6d01f","Type":"ContainerStarted","Data":"1079195b821eeeb440d66529a1b90b1f3a65971120ba4c53acb584e80b0c1f3e"}
Jul 07 16:48:05.516513 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.516460    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tvtzh" podStartSLOduration=3.647060826 podStartE2EDuration="22.516440618s" totalImagesPullingTime="18.869379792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.516329372 +0000 UTC m=+23.779866031" watchObservedRunningTime="2026-07-07 16:48:05.516440618 +0000 UTC m=+23.779977252"
Jul 07 16:48:05.533661 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.533615    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ftff6" podStartSLOduration=3.661434875 podStartE2EDuration="22.53359981s" totalImagesPullingTime="18.872164935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.533024531 +0000 UTC m=+23.796561187" watchObservedRunningTime="2026-07-07 16:48:05.53359981 +0000 UTC m=+23.797136441"
Jul 07 16:48:05.551254 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.551205    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mtp24" podStartSLOduration=3.69188297 podStartE2EDuration="22.551184815s" totalImagesPullingTime="18.859301845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.550375743 +0000 UTC m=+23.813912395" watchObservedRunningTime="2026-07-07 16:48:05.551184815 +0000 UTC m=+23.814721450"
Jul 07 16:48:05.629195 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.629134    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-64nzh" podStartSLOduration=3.693934693 podStartE2EDuration="22.629119616s" totalImagesPullingTime="18.935184923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.586425886 +0000 UTC m=+23.849962519" watchObservedRunningTime="2026-07-07 16:48:05.629119616 +0000 UTC m=+23.892656248"
Jul 07 16:48:05.757521 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.757491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:05.839134 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.839107    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:05.921577 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:05.921549    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:06.062156 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:06.062081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:06.148101 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:06.147992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:06.485772 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:06.485736    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" event={"ID":"282109b7-c9be-4d64-a867-4281aa2b16b9","Type":"ContainerStarted","Data":"87411405dc4f98141bdc4c1472ed843bae226a40fe11c18e4767478a6e9d97a5"}
Jul 07 16:48:06.486628 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:06.485870    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" event={"ID":"282109b7-c9be-4d64-a867-4281aa2b16b9","Type":"ContainerStarted","Data":"927cf6ace20d8d3d92e0f35f6316c9a9c031a4344e7d827957fe405d37992f54"}
Jul 07 16:48:06.486628 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:06.485922    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" event={"ID":"282109b7-c9be-4d64-a867-4281aa2b16b9","Type":"ContainerStarted","Data":"b718171708aa5fe0b0effee73e86d13dd5f623adfc9b4f50abb67a68b5fe3eac"}
Jul 07 16:48:06.486628 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:06.485968    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" event={"ID":"282109b7-c9be-4d64-a867-4281aa2b16b9","Type":"ContainerStarted","Data":"dd805824c963b303af7679fb583a81d965257c42a9341bbe6975def4c33d0dba"}
Jul 07 16:48:06.970309 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:06.970278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:07.071812 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.071783    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:07.120479 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.120455    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 16:48:07.172308 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.172277    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:07.190277 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.190200    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T16:48:07.120475114Z","UUID":"7c12d228-648b-4a27-bbe1-a2eec6eef3a1","Handler":null,"Name":"","Endpoint":""}
Jul 07 16:48:07.191561 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.191537    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 16:48:07.191624 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.191573    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 16:48:07.331342 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.331312    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:07.331538 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.331312    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:07.331538 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:07.331405    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:07.331643 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:07.331589    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:07.331643 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.331613    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:07.331749 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:07.331695    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:07.489585 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.489552    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m5zt9" event={"ID":"da05c267-ecba-4f27-883a-f5f9792142a0","Type":"ContainerStarted","Data":"fe056cd710f78634aa300531a9e2de1543646ebb210369641ceb01c8532d001f"}
Jul 07 16:48:07.491252 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.491220    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2" event={"ID":"dfa8416e-b75a-41d8-9d23-9d3aee893e52","Type":"ContainerStarted","Data":"fe05e50529902396cd4433791f6c2fdf66ed1fb75fea5be8ac0721b321fd7df7"}
Jul 07 16:48:07.526289 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.526250    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m5zt9" podStartSLOduration=5.645501588 podStartE2EDuration="24.526237445s" totalImagesPullingTime="18.880735857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:07.52584307 +0000 UTC m=+25.789379702" watchObservedRunningTime="2026-07-07 16:48:07.526237445 +0000 UTC m=+25.789774110"
Jul 07 16:48:07.526429 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.526403    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kz8s2" podStartSLOduration=5.662849623 podStartE2EDuration="24.526393136s" totalImagesPullingTime="18.863543513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.629202873 +0000 UTC m=+23.892739506" watchObservedRunningTime="2026-07-07 16:48:07.526393136 +0000 UTC m=+25.789929768"
Jul 07 16:48:07.829660 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.829632    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:48:07.830516 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.830499    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:48:07.884676 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:07.884644    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:08.324831 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:08.324797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:08.430414 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:08.430387    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:08.497061 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:08.497032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" event={"ID":"282109b7-c9be-4d64-a867-4281aa2b16b9","Type":"ContainerStarted","Data":"713a855bf2c24e69512c7929a42019b247a3cfa25d5cfa4c016f398fcb83c78a"}
Jul 07 16:48:08.497427 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:08.497274    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:48:08.497657 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:08.497637    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mtp24"
Jul 07 16:48:08.500475 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:08.500395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:08.572969 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:08.572938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:08.638528 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:08.638439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:09.330644 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:09.330437    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:09.330830 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:09.330439    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:09.330830 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:09.330768    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:09.330830 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:09.330444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:09.330830 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:09.330818    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:09.331064 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:09.330906    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:09.334935 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:09.334909    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:09.475451 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:09.475405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:09.501201 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:09.501166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2" event={"ID":"dfa8416e-b75a-41d8-9d23-9d3aee893e52","Type":"ContainerStarted","Data":"783501ceb61652ab5e7069674a321bd7a8f875809280458723d0605d81c0363e"}
Jul 07 16:48:09.521122 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:09.521066    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fmcd2" podStartSLOduration=4.785324566 podStartE2EDuration="26.521048319s" totalImagesPullingTime="21.735723753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:09.521042547 +0000 UTC m=+27.784579180" watchObservedRunningTime="2026-07-07 16:48:09.521048319 +0000 UTC m=+27.784584946"
Jul 07 16:48:09.545771 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:09.545743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:10.113063 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:10.112966    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:10.216241 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:10.216209    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:10.384886 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:10.384858    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:10.483610 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:10.482651    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:10.507354 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:10.507329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" event={"ID":"282109b7-c9be-4d64-a867-4281aa2b16b9","Type":"ContainerStarted","Data":"bad01c0ad4e568d2a87a7853b214e5bcdc470a0e06bb2708bc7ae1a65c7dd454"}
Jul 07 16:48:10.539739 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:10.539691    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9" podStartSLOduration=8.611459774 podStartE2EDuration="27.539675489s" totalImagesPullingTime="18.928215715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:10.538723149 +0000 UTC m=+28.802259782" watchObservedRunningTime="2026-07-07 16:48:10.539675489 +0000 UTC m=+28.803212120"
Jul 07 16:48:10.583175 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:10.583004    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:10.857517 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:10.857489    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tvtzh_67a4be0d-4845-43ec-a5d3-2952c9a6d01f/dns-node-resolver/0.log"
Jul 07 16:48:11.287938 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.287905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:11.330943 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.330916    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:11.331106 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.331030    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:11.331106 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:11.331039    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:11.331226 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:11.331123    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:11.331226 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.331167    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:11.331326 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:11.331224    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:11.510746 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.510713    2559 generic.go:356] "Generic (PLEG): container finished" podID="709bde3f-4f89-4234-8ec7-c96fcb8333f5" containerID="a86e2cf8708217927b00a7eee2b2c934692a0ca97276094862ddace7efeb1367" exitCode=0
Jul 07 16:48:11.511381 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.510794    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg2f8" event={"ID":"709bde3f-4f89-4234-8ec7-c96fcb8333f5","Type":"ContainerDied","Data":"a86e2cf8708217927b00a7eee2b2c934692a0ca97276094862ddace7efeb1367"}
Jul 07 16:48:11.511381 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.511279    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:48:11.511381 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.511364    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:48:11.512158 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.511416    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:48:11.527694 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.527671    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:48:11.528018 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.527966    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:48:11.660041 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.659948    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ftff6_4a1625b5-d332-4801-8459-07c873619a65/node-ca/0.log"
Jul 07 16:48:11.763203 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.763170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:11.838585 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:11.838543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:12.652985 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:12.652952    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c57bv"]
Jul 07 16:48:12.653320 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:12.653074    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:12.653320 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:12.653175    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:12.655615 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:12.655594    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pjtgp"]
Jul 07 16:48:12.655696 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:12.655672    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:12.655750 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:12.655735    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:12.659542 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:12.659525    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vp8v6"]
Jul 07 16:48:12.659645 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:12.659616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:12.659738 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:12.659718    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:12.672095 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:12.672069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:13.519336 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:13.519304    2559 generic.go:356] "Generic (PLEG): container finished" podID="709bde3f-4f89-4234-8ec7-c96fcb8333f5" containerID="c840b57688ba27af811de24cfe493513e7ff2a7bb33046a167e5cb0103ede586" exitCode=0
Jul 07 16:48:13.519515 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:13.519389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg2f8" event={"ID":"709bde3f-4f89-4234-8ec7-c96fcb8333f5","Type":"ContainerDied","Data":"c840b57688ba27af811de24cfe493513e7ff2a7bb33046a167e5cb0103ede586"}
Jul 07 16:48:13.810281 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:13.810197    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:13.918129 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:13.917943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:14.014740 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:14.014697    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:14.132334 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:14.132257    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:14.222200 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:14.222167    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:14.322065 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:14.322035    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:14.330949 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:14.330932    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:14.331053 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:14.331036    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:14.331136 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:14.331119    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:14.331183 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:14.331038    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:14.331183 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:14.331042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:14.331249 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:14.331197    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:14.623920 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:14.623888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:15.466553 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:15.466517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:15.525472 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:15.525440    2559 generic.go:356] "Generic (PLEG): container finished" podID="709bde3f-4f89-4234-8ec7-c96fcb8333f5" containerID="b5760a8b2e2793d212b58f3aff2333d2d00230e777a698ce6c050edff0dd94b6" exitCode=0
Jul 07 16:48:15.525602 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:15.525483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg2f8" event={"ID":"709bde3f-4f89-4234-8ec7-c96fcb8333f5","Type":"ContainerDied","Data":"b5760a8b2e2793d212b58f3aff2333d2d00230e777a698ce6c050edff0dd94b6"}
Jul 07 16:48:15.618159 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:15.618136    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:15.800034 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:15.800007    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:15.881405 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:15.881351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:16.250268 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:16.250215    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:16.331495 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:16.331419    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:16.331651 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:16.331425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:16.331651 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:16.331550    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:16.331651 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.331546    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:16.331651 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.331621    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:16.331834 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.331675    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:16.428314 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:16.428273    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:16.828252 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:16.828208    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:16.863235 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:16.863204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:16.863386 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.863322    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:48:16.863386 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.863384    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs podName:fb7c2897-e46e-456b-b3d2-7f419224c6b3 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:48.863366512 +0000 UTC m=+67.126903136 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs") pod "network-metrics-daemon-vp8v6" (UID: "fb7c2897-e46e-456b-b3d2-7f419224c6b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:48:16.964338 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:16.964308    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:16.964487 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:16.964372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:16.964487 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.964425    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:48:16.964487 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.964445    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:48:16.964487 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.964455    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k5kwd for pod openshift-network-diagnostics/network-check-target-pjtgp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:48:16.964621 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.964492    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:48:16.964621 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.964505    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd podName:077f2741-19ab-4827-ace5-c11203efc785 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:48.964491463 +0000 UTC m=+67.228028078 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k5kwd" (UniqueName: "kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd") pod "network-check-target-pjtgp" (UID: "077f2741-19ab-4827-ace5-c11203efc785") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:48:16.964621 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:16.964539    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret podName:d273870c-4ed6-44f4-be7c-669c1c013396 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:48.964524051 +0000 UTC m=+67.228060664 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret") pod "global-pull-secret-syncer-c57bv" (UID: "d273870c-4ed6-44f4-be7c-669c1c013396") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:48:17.616783 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:17.616748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:17.719110 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:17.719065    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:17.838213 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:17.838181    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:18.007650 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:18.007612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:18.112950 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:18.112914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:18.331108 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:18.331029    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:18.331108 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:18.331073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:18.331317 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:18.331036    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:18.331317 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:18.331161    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:18.331317 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:18.331236    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:18.331475 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:18.331317    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:18.417684 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:18.417653    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:18.517934 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:18.517900    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:19.022577 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:19.022540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:19.625211 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:19.625164    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:19.815370 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:19.815325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:20.017489 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:20.017454    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:20.118711 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:20.118676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:20.330617 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:20.330588    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:20.330762 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:20.330598    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:20.330762 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:20.330692    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:20.330861 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:20.330793    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:20.330861 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:20.330599    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:20.330929 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:20.330894    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:20.413201 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:20.413158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:20.538926 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:20.538850    2559 generic.go:356] "Generic (PLEG): container finished" podID="709bde3f-4f89-4234-8ec7-c96fcb8333f5" containerID="ca84f6622cddd74ad856c71c456c0425dd0bdcfc01c1d7ecc0e9515fcf2e2d41" exitCode=0
Jul 07 16:48:20.538926 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:20.538908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg2f8" event={"ID":"709bde3f-4f89-4234-8ec7-c96fcb8333f5","Type":"ContainerDied","Data":"ca84f6622cddd74ad856c71c456c0425dd0bdcfc01c1d7ecc0e9515fcf2e2d41"}
Jul 07 16:48:21.015268 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:21.015238    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:21.543482 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:21.543446    2559 generic.go:356] "Generic (PLEG): container finished" podID="709bde3f-4f89-4234-8ec7-c96fcb8333f5" containerID="3533dd1092cea7ff4e056d22ce645677f3cff9fd16386800ce7802163e17d638" exitCode=0
Jul 07 16:48:21.543910 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:21.543525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg2f8" event={"ID":"709bde3f-4f89-4234-8ec7-c96fcb8333f5","Type":"ContainerDied","Data":"3533dd1092cea7ff4e056d22ce645677f3cff9fd16386800ce7802163e17d638"}
Jul 07 16:48:21.635412 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:21.635247    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:21.810675 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:21.810596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:22.022198 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:22.022163    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:22.331160 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:22.331127    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:22.331342 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:22.331259    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:22.331342 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:22.331262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:22.331342 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:22.331304    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:22.331512 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:22.331404    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:22.331567 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:22.331508    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:22.410311 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:22.410279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:22.549049 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:22.549015    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg2f8" event={"ID":"709bde3f-4f89-4234-8ec7-c96fcb8333f5","Type":"ContainerStarted","Data":"0d4c7969cd8ef80c5453309436b6a551dfe7dc6018bc28206147b6a09dbce84a"}
Jul 07 16:48:22.574172 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:22.574127    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gg2f8" podStartSLOduration=8.991084914 podStartE2EDuration="39.574115285s" totalImagesPullingTime="30.583030371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:22.573767322 +0000 UTC m=+40.837303953" watchObservedRunningTime="2026-07-07 16:48:22.574115285 +0000 UTC m=+40.837651918"
Jul 07 16:48:22.737606 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:22.737538    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:22.839329 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:22.839284    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:23.012573 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:23.012542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:23.420704 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:23.420618    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:23.811993 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:23.811951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:23.914238 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:23.914187    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:24.035523 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:24.035490    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:24.217463 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:24.217379    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:24.330486 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:24.330455    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:24.330657 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:24.330572    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:24.330657 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:24.330583    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:24.330657 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:24.330607    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:24.330764 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:24.330689    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:24.330799 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:24.330775    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:24.416536 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:24.416506    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:24.619054 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:24.619021    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:24.949733 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:24.949649    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:26.330834 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:26.330801    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:26.331320 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:26.330801    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:26.331320 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:26.330917    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:26.331320 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:26.330908    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:26.331320 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:26.331002    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:26.331320 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:26.331068    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:28.331130 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:28.331101    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:28.331130 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:28.331131    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:28.331602 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:28.331192    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:28.331602 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:28.331216    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:28.331602 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:28.331288    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:28.331602 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:28.331363    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:30.330491 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:30.330461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:30.330864 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:30.330461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:30.330864 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:30.330559    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c57bv" podUID="d273870c-4ed6-44f4-be7c-669c1c013396"
Jul 07 16:48:30.330864 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:30.330461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:30.330864 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:30.330650    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vp8v6" podUID="fb7c2897-e46e-456b-b3d2-7f419224c6b3"
Jul 07 16:48:30.330864 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:30.330711    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjtgp" podUID="077f2741-19ab-4827-ace5-c11203efc785"
Jul 07 16:48:30.801090 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:30.801051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:30.914635 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:30.914594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:31.237996 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:31.237902    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:31.386014 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:31.385967    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:31.509048 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:31.509019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:31.608815 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:31.608780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:31.974505 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:31.974437    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-63.ec2.internal" event="NodeReady"
Jul 07 16:48:31.974643 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:31.974558    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 16:48:32.029744 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.029718    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d8f87c964-kkjnq"]
Jul 07 16:48:32.042904 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.042880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.045249 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.045227    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 16:48:32.045356 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.045277    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 16:48:32.045542 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.045529    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pk9r6\""
Jul 07 16:48:32.045875 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.045860    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 16:48:32.051324 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.051303    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 16:48:32.054376 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.054356    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cwz47"]
Jul 07 16:48:32.073899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.073881    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-twsng"]
Jul 07 16:48:32.074023 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.074008    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.077430 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.077409    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 16:48:32.077587 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.077488    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 16:48:32.077704 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.077493    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 16:48:32.077704 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.077502    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 16:48:32.077704 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.077533    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5tvm4\""
Jul 07 16:48:32.082235 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1321693f-e32d-4e48-9754-bceb0ec4431d-trusted-ca\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.082317 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxt8\" (UniqueName: \"kubernetes.io/projected/1321693f-e32d-4e48-9754-bceb0ec4431d-kube-api-access-hmxt8\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.082317 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1321693f-e32d-4e48-9754-bceb0ec4431d-bound-sa-token\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.082317 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082290    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1321693f-e32d-4e48-9754-bceb0ec4431d-image-registry-private-configuration\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.082317 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpbk\" (UniqueName: \"kubernetes.io/projected/f92b530c-2cf9-4e35-81a8-be8db3daaf48-kube-api-access-xxpbk\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.082474 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f92b530c-2cf9-4e35-81a8-be8db3daaf48-crio-socket\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.082474 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1321693f-e32d-4e48-9754-bceb0ec4431d-ca-trust-extracted\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.082474 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1321693f-e32d-4e48-9754-bceb0ec4431d-installation-pull-secrets\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.082474 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f92b530c-2cf9-4e35-81a8-be8db3daaf48-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.082474 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f92b530c-2cf9-4e35-81a8-be8db3daaf48-data-volume\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.082474 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082421    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1321693f-e32d-4e48-9754-bceb0ec4431d-registry-tls\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.082665 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1321693f-e32d-4e48-9754-bceb0ec4431d-registry-certificates\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.082665 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.082518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f92b530c-2cf9-4e35-81a8-be8db3daaf48-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.087210 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.087193    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d8f87c964-kkjnq"]
Jul 07 16:48:32.087281 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.087219    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-twsng"]
Jul 07 16:48:32.087281 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.087237    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cwz47"]
Jul 07 16:48:32.087346 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.087307    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-twsng"
Jul 07 16:48:32.089706 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.089688    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 16:48:32.089805 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.089789    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 16:48:32.090223 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.090209    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 16:48:32.091686 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.091669    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g82pc\""
Jul 07 16:48:32.151895 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.151868    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4lhzc"]
Jul 07 16:48:32.156343 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.156317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:32.170817 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.170794    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4lhzc"]
Jul 07 16:48:32.170902 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.170872    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.173787 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.173767    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 16:48:32.174139 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.174122    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 16:48:32.174209 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.174128    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cn8rd\""
Jul 07 16:48:32.182733 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.182715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5409188f-4647-4517-a322-06c705e56fff-metrics-tls\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.182835 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.182740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdl7m\" (UniqueName: \"kubernetes.io/projected/1b7c6701-1f14-4950-9d40-25e7c24fd3b4-kube-api-access-vdl7m\") pod \"ingress-canary-twsng\" (UID: \"1b7c6701-1f14-4950-9d40-25e7c24fd3b4\") " pod="openshift-ingress-canary/ingress-canary-twsng"
Jul 07 16:48:32.182835 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.182763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1321693f-e32d-4e48-9754-bceb0ec4431d-ca-trust-extracted\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.182835 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.182778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1321693f-e32d-4e48-9754-bceb0ec4431d-installation-pull-secrets\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.182835 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.182794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f92b530c-2cf9-4e35-81a8-be8db3daaf48-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.183056 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.182835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdpqh\" (UniqueName: \"kubernetes.io/projected/5409188f-4647-4517-a322-06c705e56fff-kube-api-access-bdpqh\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.183056 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.182960    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f92b530c-2cf9-4e35-81a8-be8db3daaf48-data-volume\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.183056 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.182999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1321693f-e32d-4e48-9754-bceb0ec4431d-registry-tls\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.183056 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1321693f-e32d-4e48-9754-bceb0ec4431d-registry-certificates\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.183224 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183063    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f92b530c-2cf9-4e35-81a8-be8db3daaf48-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.183224 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1321693f-e32d-4e48-9754-bceb0ec4431d-trusted-ca\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.183224 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183105    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxt8\" (UniqueName: \"kubernetes.io/projected/1321693f-e32d-4e48-9754-bceb0ec4431d-kube-api-access-hmxt8\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.183224 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1321693f-e32d-4e48-9754-bceb0ec4431d-bound-sa-token\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.183224 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183175    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409188f-4647-4517-a322-06c705e56fff-tmp-dir\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.183224 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5409188f-4647-4517-a322-06c705e56fff-config-volume\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.183484 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1321693f-e32d-4e48-9754-bceb0ec4431d-image-registry-private-configuration\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.183484 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183254    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f92b530c-2cf9-4e35-81a8-be8db3daaf48-data-volume\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.183484 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpbk\" (UniqueName: \"kubernetes.io/projected/f92b530c-2cf9-4e35-81a8-be8db3daaf48-kube-api-access-xxpbk\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.183484 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183283    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b7c6701-1f14-4950-9d40-25e7c24fd3b4-cert\") pod \"ingress-canary-twsng\" (UID: \"1b7c6701-1f14-4950-9d40-25e7c24fd3b4\") " pod="openshift-ingress-canary/ingress-canary-twsng"
Jul 07 16:48:32.183484 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f92b530c-2cf9-4e35-81a8-be8db3daaf48-crio-socket\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.183484 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183425    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f92b530c-2cf9-4e35-81a8-be8db3daaf48-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.183747 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183501    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f92b530c-2cf9-4e35-81a8-be8db3daaf48-crio-socket\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.183747 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.183610    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1321693f-e32d-4e48-9754-bceb0ec4431d-ca-trust-extracted\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.184204 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.184055    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1321693f-e32d-4e48-9754-bceb0ec4431d-registry-certificates\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.184322 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.184301    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1321693f-e32d-4e48-9754-bceb0ec4431d-trusted-ca\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.186946 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.186925    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f92b530c-2cf9-4e35-81a8-be8db3daaf48-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.187060 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.187040    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1321693f-e32d-4e48-9754-bceb0ec4431d-image-registry-private-configuration\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.187101 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.187040    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1321693f-e32d-4e48-9754-bceb0ec4431d-installation-pull-secrets\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.187202 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.187181    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1321693f-e32d-4e48-9754-bceb0ec4431d-registry-tls\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.195896 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.195876    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpbk\" (UniqueName: \"kubernetes.io/projected/f92b530c-2cf9-4e35-81a8-be8db3daaf48-kube-api-access-xxpbk\") pod \"insights-runtime-extractor-cwz47\" (UID: \"f92b530c-2cf9-4e35-81a8-be8db3daaf48\") " pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.197283 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.197264    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxt8\" (UniqueName: \"kubernetes.io/projected/1321693f-e32d-4e48-9754-bceb0ec4431d-kube-api-access-hmxt8\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.200602 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.200583    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1321693f-e32d-4e48-9754-bceb0ec4431d-bound-sa-token\") pod \"image-registry-d8f87c964-kkjnq\" (UID: \"1321693f-e32d-4e48-9754-bceb0ec4431d\") " pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.283890 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.283864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5409188f-4647-4517-a322-06c705e56fff-metrics-tls\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.284034 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.283897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdl7m\" (UniqueName: \"kubernetes.io/projected/1b7c6701-1f14-4950-9d40-25e7c24fd3b4-kube-api-access-vdl7m\") pod \"ingress-canary-twsng\" (UID: \"1b7c6701-1f14-4950-9d40-25e7c24fd3b4\") " pod="openshift-ingress-canary/ingress-canary-twsng"
Jul 07 16:48:32.284034 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.283923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdpqh\" (UniqueName: \"kubernetes.io/projected/5409188f-4647-4517-a322-06c705e56fff-kube-api-access-bdpqh\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.284159 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.284087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409188f-4647-4517-a322-06c705e56fff-tmp-dir\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.284159 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.284119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5409188f-4647-4517-a322-06c705e56fff-config-volume\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.284159 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.284138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b7c6701-1f14-4950-9d40-25e7c24fd3b4-cert\") pod \"ingress-canary-twsng\" (UID: \"1b7c6701-1f14-4950-9d40-25e7c24fd3b4\") " pod="openshift-ingress-canary/ingress-canary-twsng"
Jul 07 16:48:32.284444 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.284415    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5409188f-4647-4517-a322-06c705e56fff-tmp-dir\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.284653 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.284637    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5409188f-4647-4517-a322-06c705e56fff-config-volume\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.286003 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.285966    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5409188f-4647-4517-a322-06c705e56fff-metrics-tls\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.286180 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.286166    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b7c6701-1f14-4950-9d40-25e7c24fd3b4-cert\") pod \"ingress-canary-twsng\" (UID: \"1b7c6701-1f14-4950-9d40-25e7c24fd3b4\") " pod="openshift-ingress-canary/ingress-canary-twsng"
Jul 07 16:48:32.288148 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.288128    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:32.297900 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.297879    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdpqh\" (UniqueName: \"kubernetes.io/projected/5409188f-4647-4517-a322-06c705e56fff-kube-api-access-bdpqh\") pod \"dns-default-4lhzc\" (UID: \"5409188f-4647-4517-a322-06c705e56fff\") " pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.298599 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.298583    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdl7m\" (UniqueName: \"kubernetes.io/projected/1b7c6701-1f14-4950-9d40-25e7c24fd3b4-kube-api-access-vdl7m\") pod \"ingress-canary-twsng\" (UID: \"1b7c6701-1f14-4950-9d40-25e7c24fd3b4\") " pod="openshift-ingress-canary/ingress-canary-twsng"
Jul 07 16:48:32.330879 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.330858    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:32.331043 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.330945    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:32.331102 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.330950    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:32.333305 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.333285    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:48:32.333387 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.333332    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:48:32.333458 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.333428    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:48:32.333518 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.333457    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:48:32.333561 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.333541    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwbcf\""
Jul 07 16:48:32.333597 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.333576    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sk64f\""
Jul 07 16:48:32.353219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.353204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:32.382300 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.382279    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cwz47"
Jul 07 16:48:32.388460 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.388435    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:32.395227 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.395204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-twsng"
Jul 07 16:48:32.480613 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.480289    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:32.507694 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.507654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:32.584260 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.584229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cwz47"]
Jul 07 16:48:32.588194 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.588123    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:32.595143 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.595112    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d8f87c964-kkjnq"]
Jul 07 16:48:32.603695 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:32.603669    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1321693f_e32d_4e48_9754_bceb0ec4431d.slice/crio-f3e05678ca304c746ed24612a9e2b17a0e6b15285c0341b9d82156a23a99d2cc WatchSource:0}: Error finding container f3e05678ca304c746ed24612a9e2b17a0e6b15285c0341b9d82156a23a99d2cc: Status 404 returned error can't find the container with id f3e05678ca304c746ed24612a9e2b17a0e6b15285c0341b9d82156a23a99d2cc
Jul 07 16:48:32.613881 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.613859    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-twsng"]
Jul 07 16:48:32.616600 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:32.616571    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7c6701_1f14_4950_9d40_25e7c24fd3b4.slice/crio-086ce1dbed33dabf8e6efe76f30748aee19dd2786c706d2f400c4d94710d4d93 WatchSource:0}: Error finding container 086ce1dbed33dabf8e6efe76f30748aee19dd2786c706d2f400c4d94710d4d93: Status 404 returned error can't find the container with id 086ce1dbed33dabf8e6efe76f30748aee19dd2786c706d2f400c4d94710d4d93
Jul 07 16:48:32.650169 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:32.650142    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4lhzc"]
Jul 07 16:48:32.653130 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:32.653104    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5409188f_4647_4517_a322_06c705e56fff.slice/crio-5721c111c4c872e96a5e85f4d9b048ffdff71f36aff98b4da005a13b3a7e197b WatchSource:0}: Error finding container 5721c111c4c872e96a5e85f4d9b048ffdff71f36aff98b4da005a13b3a7e197b: Status 404 returned error can't find the container with id 5721c111c4c872e96a5e85f4d9b048ffdff71f36aff98b4da005a13b3a7e197b
Jul 07 16:48:33.111227 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.111186    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-fjddn"]
Jul 07 16:48:33.125543 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.125433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:33.127199 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.127153    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-fjddn"]
Jul 07 16:48:33.127378 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.127266    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-fjddn"
Jul 07 16:48:33.133390 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.133177    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 16:48:33.133892 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.133610    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 16:48:33.135065 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.134256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dr4j8\""
Jul 07 16:48:33.189944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.189734    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwsq\" (UniqueName: \"kubernetes.io/projected/232694f9-b34b-49bd-b61a-e72e39479653-kube-api-access-dxwsq\") pod \"downloads-685bfb5f64-fjddn\" (UID: \"232694f9-b34b-49bd-b61a-e72e39479653\") " pod="openshift-console/downloads-685bfb5f64-fjddn"
Jul 07 16:48:33.223604 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.223542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:33.290517 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.290447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwsq\" (UniqueName: \"kubernetes.io/projected/232694f9-b34b-49bd-b61a-e72e39479653-kube-api-access-dxwsq\") pod \"downloads-685bfb5f64-fjddn\" (UID: \"232694f9-b34b-49bd-b61a-e72e39479653\") " pod="openshift-console/downloads-685bfb5f64-fjddn"
Jul 07 16:48:33.301698 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.301669    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwsq\" (UniqueName: \"kubernetes.io/projected/232694f9-b34b-49bd-b61a-e72e39479653-kube-api-access-dxwsq\") pod \"downloads-685bfb5f64-fjddn\" (UID: \"232694f9-b34b-49bd-b61a-e72e39479653\") " pod="openshift-console/downloads-685bfb5f64-fjddn"
Jul 07 16:48:33.332466 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.332438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:33.426936 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.426831    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:33.440028 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.439999    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-fjddn"
Jul 07 16:48:33.581519 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.581429    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-twsng" event={"ID":"1b7c6701-1f14-4950-9d40-25e7c24fd3b4","Type":"ContainerStarted","Data":"086ce1dbed33dabf8e6efe76f30748aee19dd2786c706d2f400c4d94710d4d93"}
Jul 07 16:48:33.585642 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.585576    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cwz47" event={"ID":"f92b530c-2cf9-4e35-81a8-be8db3daaf48","Type":"ContainerStarted","Data":"186dcbf27df72071cf0e8f8553d362b41bf4c520733081894f0341f6c3a13077"}
Jul 07 16:48:33.591783 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.591014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d8f87c964-kkjnq" event={"ID":"1321693f-e32d-4e48-9754-bceb0ec4431d","Type":"ContainerStarted","Data":"93572ebeaa1eac20ab302e17455269727fd9b7e4da0313d6a98887f168204657"}
Jul 07 16:48:33.591783 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.591060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d8f87c964-kkjnq" event={"ID":"1321693f-e32d-4e48-9754-bceb0ec4431d","Type":"ContainerStarted","Data":"f3e05678ca304c746ed24612a9e2b17a0e6b15285c0341b9d82156a23a99d2cc"}
Jul 07 16:48:33.591783 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.591708    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:33.593562 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.593479    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4lhzc" event={"ID":"5409188f-4647-4517-a322-06c705e56fff","Type":"ContainerStarted","Data":"5721c111c4c872e96a5e85f4d9b048ffdff71f36aff98b4da005a13b3a7e197b"}
Jul 07 16:48:33.606351 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.606290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:33.614831 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.614396    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d8f87c964-kkjnq" podStartSLOduration=2.614377132 podStartE2EDuration="2.614377132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:33.612425066 +0000 UTC m=+51.875961721" watchObservedRunningTime="2026-07-07 16:48:33.614377132 +0000 UTC m=+51.877913766"
Jul 07 16:48:33.638010 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.637969    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-fjddn"]
Jul 07 16:48:33.642605 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:33.642563    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232694f9_b34b_49bd_b61a_e72e39479653.slice/crio-367b8a7451c6876cf87f2f54aee7d45ec7c085675878775113ece60dafea7576 WatchSource:0}: Error finding container 367b8a7451c6876cf87f2f54aee7d45ec7c085675878775113ece60dafea7576: Status 404 returned error can't find the container with id 367b8a7451c6876cf87f2f54aee7d45ec7c085675878775113ece60dafea7576
Jul 07 16:48:33.705960 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.705878    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:33.995111 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:33.994962    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:34.093669 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:34.093640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:34.206242 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:34.206172    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:34.598408 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:34.598310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-fjddn" event={"ID":"232694f9-b34b-49bd-b61a-e72e39479653","Type":"ContainerStarted","Data":"367b8a7451c6876cf87f2f54aee7d45ec7c085675878775113ece60dafea7576"}
Jul 07 16:48:34.615524 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:34.615489    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:34.808707 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:34.808668    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:34.908608 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:34.908517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:35.824068 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:35.824035    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:35.889493 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:35.889458    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:36.614608 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:36.614571    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:36.716235 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:36.716193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:36.796238 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:36.796208    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:36.979908 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:36.979781    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:39.617229 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:39.617188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4lhzc" event={"ID":"5409188f-4647-4517-a322-06c705e56fff","Type":"ContainerStarted","Data":"ac128b6e1ed50b55e0c03d7da67ef2141c115c4566ae07b75074d7cd618eb8f9"}
Jul 07 16:48:39.617741 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:39.617257    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4lhzc" event={"ID":"5409188f-4647-4517-a322-06c705e56fff","Type":"ContainerStarted","Data":"13eb8703eb5ccb17273b337d1d4123236ff0f529f551f922c0fea9d6f5824e95"}
Jul 07 16:48:39.617741 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:39.617402    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:39.619246 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:39.619205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-twsng" event={"ID":"1b7c6701-1f14-4950-9d40-25e7c24fd3b4","Type":"ContainerStarted","Data":"184239531a4efdbf2ce1f58739394f85c5d4f3a89c7650cf91674ea82e186f43"}
Jul 07 16:48:39.620846 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:39.620821    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cwz47" event={"ID":"f92b530c-2cf9-4e35-81a8-be8db3daaf48","Type":"ContainerStarted","Data":"620216137cb416c367aa7ebdfd287e0641dbf89fd2564d02a14cb3444be10d33"}
Jul 07 16:48:39.638811 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:39.638760    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4lhzc" podStartSLOduration=1.5362601599999999 podStartE2EDuration="7.638742436s" totalImagesPullingTime="6.102482276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:39.63775841 +0000 UTC m=+57.901295043" watchObservedRunningTime="2026-07-07 16:48:39.638742436 +0000 UTC m=+57.902279047"
Jul 07 16:48:39.657869 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:39.657813    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-twsng" podStartSLOduration=1.5188082870000001 podStartE2EDuration="7.657801188s" totalImagesPullingTime="6.138992901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:39.655765674 +0000 UTC m=+57.919302318" watchObservedRunningTime="2026-07-07 16:48:39.657801188 +0000 UTC m=+57.921337820"
Jul 07 16:48:39.832460 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:39.832430    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:39.918572 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:39.918542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:40.074178 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:40.074122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:40.437712 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:40.437672    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:40.626196 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:40.626158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cwz47" event={"ID":"f92b530c-2cf9-4e35-81a8-be8db3daaf48","Type":"ContainerStarted","Data":"8a5afef43d19637d31b711f2b77e05e26b4865085904096b07572dc0dd3321a1"}
Jul 07 16:48:42.631290 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:42.631229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:42.732573 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:42.732539    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:42.876730 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:42.876628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:43.538254 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:43.538222    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkhn9"
Jul 07 16:48:43.642008 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:43.641924    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:43.643338 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:43.643307    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cwz47" event={"ID":"f92b530c-2cf9-4e35-81a8-be8db3daaf48","Type":"ContainerStarted","Data":"47f1771fd2b8d278b0f579448ac2262ea474ca9998bd7a7f64fe5c0dd9976c28"}
Jul 07 16:48:43.665727 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:43.665671    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cwz47" podStartSLOduration=0.95789247 podStartE2EDuration="11.665654291s" totalImagesPullingTime="10.707761821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:48:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:43.664281333 +0000 UTC m=+61.927817977" watchObservedRunningTime="2026-07-07 16:48:43.665654291 +0000 UTC m=+61.929190926"
Jul 07 16:48:43.743438 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:43.743358    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:43.843126 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:43.843085    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:43.947816 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:43.947779    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:45.002588 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.002553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:45.234925 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.234891    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"]
Jul 07 16:48:45.237936 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.237912    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.240748 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.240721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fwcz8\""
Jul 07 16:48:45.240845 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.240746    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 16:48:45.241526 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.241505    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 16:48:45.241637 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.241529    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 16:48:45.241637 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.241530    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 16:48:45.241637 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.241515    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 16:48:45.250168 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.250145    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"]
Jul 07 16:48:45.255963 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.255942    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8s562"]
Jul 07 16:48:45.258480 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.258371    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.259677 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.259641    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"]
Jul 07 16:48:45.261065 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.261044    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 16:48:45.261179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.261154    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5gvkf\""
Jul 07 16:48:45.261374 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.261356    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 16:48:45.261499 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.261479    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 16:48:45.261895 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.261878    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.263826 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.263781    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 16:48:45.263954 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.263855    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 16:48:45.264279 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.264261    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dmnh9\""
Jul 07 16:48:45.264364 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.264265    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 16:48:45.278752 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.278731    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"]
Jul 07 16:48:45.293169 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293143    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afc81d20-2e49-4284-96ed-866c03196411-sys\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.293266 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293177    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzfw\" (UniqueName: \"kubernetes.io/projected/81b2546a-97bc-4af2-b109-5f05f40f8e23-kube-api-access-7nzfw\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.293266 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhm7k\" (UniqueName: \"kubernetes.io/projected/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-api-access-lhm7k\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.293495 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b2546a-97bc-4af2-b109-5f05f40f8e23-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.293495 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc81d20-2e49-4284-96ed-866c03196411-metrics-client-ca\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.293495 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293334    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.293495 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.293495 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.293495 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a2918-c0c2-4900-a335-ccb31870a05b-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.293776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293492    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb0a2918-c0c2-4900-a335-ccb31870a05b-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.293776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81b2546a-97bc-4af2-b109-5f05f40f8e23-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.293776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-textfile\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.293776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-accelerators-collector-config\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.293776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82pjm\" (UniqueName: \"kubernetes.io/projected/afc81d20-2e49-4284-96ed-866c03196411-kube-api-access-82pjm\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.293776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293660    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-tls\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.293776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293741    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.294036 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293777    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afc81d20-2e49-4284-96ed-866c03196411-root\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.294036 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-wtmp\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.294036 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.293851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81b2546a-97bc-4af2-b109-5f05f40f8e23-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.337023 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.336926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:45.395017 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.394955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb0a2918-c0c2-4900-a335-ccb31870a05b-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.395181 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81b2546a-97bc-4af2-b109-5f05f40f8e23-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.395181 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-textfile\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395181 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-accelerators-collector-config\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395181 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82pjm\" (UniqueName: \"kubernetes.io/projected/afc81d20-2e49-4284-96ed-866c03196411-kube-api-access-82pjm\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395181 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-tls\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395425 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.395425 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395247    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afc81d20-2e49-4284-96ed-866c03196411-root\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395425 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395279    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-wtmp\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395425 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81b2546a-97bc-4af2-b109-5f05f40f8e23-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.395425 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:45.395356    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 16:48:45.395425 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395399    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afc81d20-2e49-4284-96ed-866c03196411-sys\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395425 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395359    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afc81d20-2e49-4284-96ed-866c03196411-sys\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395425 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:45.395423    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-tls podName:afc81d20-2e49-4284-96ed-866c03196411 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:45.895402757 +0000 UTC m=+64.158939383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-tls") pod "node-exporter-8s562" (UID: "afc81d20-2e49-4284-96ed-866c03196411") : secret "node-exporter-tls" not found
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395440    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb0a2918-c0c2-4900-a335-ccb31870a05b-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395441    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-textfile\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:45.395489    2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395535    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afc81d20-2e49-4284-96ed-866c03196411-root\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395551    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-wtmp\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:48:45.395582    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-tls podName:eb0a2918-c0c2-4900-a335-ccb31870a05b nodeName:}" failed. No retries permitted until 2026-07-07 16:48:45.895564394 +0000 UTC m=+64.159101005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-gfdb6" (UID: "eb0a2918-c0c2-4900-a335-ccb31870a05b") : secret "kube-state-metrics-tls" not found
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzfw\" (UniqueName: \"kubernetes.io/projected/81b2546a-97bc-4af2-b109-5f05f40f8e23-kube-api-access-7nzfw\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhm7k\" (UniqueName: \"kubernetes.io/projected/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-api-access-lhm7k\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b2546a-97bc-4af2-b109-5f05f40f8e23-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc81d20-2e49-4284-96ed-866c03196411-metrics-client-ca\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.395792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.396347 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.396347 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.396347 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a2918-c0c2-4900-a335-ccb31870a05b-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.396347 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.395967    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-accelerators-collector-config\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.396556 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.396353    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afc81d20-2e49-4284-96ed-866c03196411-metrics-client-ca\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.396556 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.396514    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b2546a-97bc-4af2-b109-5f05f40f8e23-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.396653 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.396604    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a2918-c0c2-4900-a335-ccb31870a05b-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.397044 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.397003    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.398312 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.398287    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81b2546a-97bc-4af2-b109-5f05f40f8e23-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.398423 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.398332    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81b2546a-97bc-4af2-b109-5f05f40f8e23-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.398957 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.398741    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.398957 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.398920    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.410832 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.410805    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82pjm\" (UniqueName: \"kubernetes.io/projected/afc81d20-2e49-4284-96ed-866c03196411-kube-api-access-82pjm\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.411729 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.411710    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzfw\" (UniqueName: \"kubernetes.io/projected/81b2546a-97bc-4af2-b109-5f05f40f8e23-kube-api-access-7nzfw\") pod \"openshift-state-metrics-54cf96dfcf-mkn4w\" (UID: \"81b2546a-97bc-4af2-b109-5f05f40f8e23\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.411828 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.411792    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhm7k\" (UniqueName: \"kubernetes.io/projected/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-api-access-lhm7k\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.430251 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.430224    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:45.529971 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.529878    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:45.549005 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.548959    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"
Jul 07 16:48:45.768248 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.768165    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:45.869002 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.868830    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:45.899899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.899872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-tls\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.900052 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.899912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.902323 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.902296    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afc81d20-2e49-4284-96ed-866c03196411-node-exporter-tls\") pod \"node-exporter-8s562\" (UID: \"afc81d20-2e49-4284-96ed-866c03196411\") " pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:45.902443 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.902349    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb0a2918-c0c2-4900-a335-ccb31870a05b-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gfdb6\" (UID: \"eb0a2918-c0c2-4900-a335-ccb31870a05b\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:45.961472 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:45.961436    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:46.170457 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:46.170379    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8s562"
Jul 07 16:48:46.175320 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:46.175292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"
Jul 07 16:48:46.238241 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:46.238198    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:46.415274 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:46.415236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:46.506117 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:46.506083    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:47.335621 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.335587    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79c9458997-rrnwn"]
Jul 07 16:48:47.338432 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.338410    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.341109 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.340936    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 16:48:47.341109 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.341010    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1qrmfqp9hjb74\""
Jul 07 16:48:47.341109 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.341025    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9kxbb\""
Jul 07 16:48:47.341109 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.341043    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 16:48:47.341393 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.341121    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 16:48:47.341703 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.341685    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 16:48:47.341790 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.341688    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 16:48:47.350885 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.350865    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79c9458997-rrnwn"]
Jul 07 16:48:47.412249 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.412214    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.412410 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.412281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.412410 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.412341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-metrics-client-ca\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.412410 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.412404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-grpc-tls\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.412555 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.412489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs68s\" (UniqueName: \"kubernetes.io/projected/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-kube-api-access-bs68s\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.412555 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.412517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-tls\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.412555 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.412540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.412704 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.412564    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.513523 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.513488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-tls\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.513696 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.513531    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.513696 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.513560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.513696 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.513606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.513696 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.513654    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.513954 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.513706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-metrics-client-ca\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.513954 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.513767    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-grpc-tls\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.513954 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.513841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs68s\" (UniqueName: \"kubernetes.io/projected/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-kube-api-access-bs68s\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.514916 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.514864    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-metrics-client-ca\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.516875 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.516813    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.517083 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.517034    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-tls\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.517263 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.517241    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.517499 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.517477    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-grpc-tls\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.517557 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.517537    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.517630 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.517611    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.522421 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.522399    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs68s\" (UniqueName: \"kubernetes.io/projected/17f6c6b7-c8eb-4101-bdcc-bde59e056f86-kube-api-access-bs68s\") pod \"thanos-querier-79c9458997-rrnwn\" (UID: \"17f6c6b7-c8eb-4101-bdcc-bde59e056f86\") " pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.648792 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.648670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:47.648971 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.648914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:48:47.720279 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:47.720235    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:48.209556 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:48.209526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:48.284226 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:48.284187    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:48.404257 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:48.404221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:48.626276 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:48.626239    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:48.927393 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:48.927303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:48.930020 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:48.929997    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:48:48.940593 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:48.940567    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7c2897-e46e-456b-b3d2-7f419224c6b3-metrics-certs\") pod \"network-metrics-daemon-vp8v6\" (UID: \"fb7c2897-e46e-456b-b3d2-7f419224c6b3\") " pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:49.028447 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.028411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:49.028613 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.028466    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:49.031059 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.031037    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:48:49.031593 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.031575    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:48:49.041588 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.041569    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:48:49.041713 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.041694    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d273870c-4ed6-44f4-be7c-669c1c013396-original-pull-secret\") pod \"global-pull-secret-syncer-c57bv\" (UID: \"d273870c-4ed6-44f4-be7c-669c1c013396\") " pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:49.052125 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.052106    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kwd\" (UniqueName: \"kubernetes.io/projected/077f2741-19ab-4827-ace5-c11203efc785-kube-api-access-k5kwd\") pod \"network-check-target-pjtgp\" (UID: \"077f2741-19ab-4827-ace5-c11203efc785\") " pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:49.142556 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.142517    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwbcf\""
Jul 07 16:48:49.148938 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.148912    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sk64f\""
Jul 07 16:48:49.150707 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.150686    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:48:49.150816 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.150794    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c57bv"
Jul 07 16:48:49.156806 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.156785    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vp8v6"
Jul 07 16:48:49.629077 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:49.629047    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4lhzc"
Jul 07 16:48:50.013474 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.013435    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:50.021013 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.020969    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"]
Jul 07 16:48:50.025481 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.025460    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:48:50.028202 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.028172    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-st594\""
Jul 07 16:48:50.028404 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.028381    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 16:48:50.032215 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.032176    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"]
Jul 07 16:48:50.086400 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.086366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:50.136625 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.136585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6nxx2\" (UID: \"332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:48:50.136786 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.136640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ncj\" (UniqueName: \"kubernetes.io/projected/332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e-kube-api-access-74ncj\") pod \"monitoring-plugin-d5dcff7cf-6nxx2\" (UID: \"332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:48:50.237791 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.237756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6nxx2\" (UID: \"332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:48:50.237971 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.237816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ncj\" (UniqueName: \"kubernetes.io/projected/332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e-kube-api-access-74ncj\") pod \"monitoring-plugin-d5dcff7cf-6nxx2\" (UID: \"332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:48:50.240769 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.240736    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6nxx2\" (UID: \"332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:48:50.246882 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.246855    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ncj\" (UniqueName: \"kubernetes.io/projected/332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e-kube-api-access-74ncj\") pod \"monitoring-plugin-d5dcff7cf-6nxx2\" (UID: \"332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:48:50.251854 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.251821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:50.337262 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.337176    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:48:50.868563 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.868522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:50.966701 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:50.966641    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:51.040709 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:51.040673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:51.515374 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:51.515335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:51.737126 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:51.737089    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc81d20_2e49_4284_96ed_866c03196411.slice/crio-e6b2faa554c1957e29b26f69cb0b4859c6fd3149ae887cecd587e013c310a22e WatchSource:0}: Error finding container e6b2faa554c1957e29b26f69cb0b4859c6fd3149ae887cecd587e013c310a22e: Status 404 returned error can't find the container with id e6b2faa554c1957e29b26f69cb0b4859c6fd3149ae887cecd587e013c310a22e
Jul 07 16:48:51.950112 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:51.950083    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6"]
Jul 07 16:48:51.950441 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:51.950335    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0a2918_c0c2_4900_a335_ccb31870a05b.slice/crio-bc742470b93585c738f147ae2d7a88437dc775deb3182aba0f40249bc1976cc9 WatchSource:0}: Error finding container bc742470b93585c738f147ae2d7a88437dc775deb3182aba0f40249bc1976cc9: Status 404 returned error can't find the container with id bc742470b93585c738f147ae2d7a88437dc775deb3182aba0f40249bc1976cc9
Jul 07 16:48:51.959100 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:51.959076    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b2546a_97bc_4af2_b109_5f05f40f8e23.slice/crio-9c6bf35e4953eb9b51fbfc0c66c3aa9e5f23665b15b12de692da2b256432dbdb WatchSource:0}: Error finding container 9c6bf35e4953eb9b51fbfc0c66c3aa9e5f23665b15b12de692da2b256432dbdb: Status 404 returned error can't find the container with id 9c6bf35e4953eb9b51fbfc0c66c3aa9e5f23665b15b12de692da2b256432dbdb
Jul 07 16:48:51.963085 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:51.963063    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w"]
Jul 07 16:48:52.174951 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.174882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c57bv"]
Jul 07 16:48:52.178755 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.178729    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vp8v6"]
Jul 07 16:48:52.180121 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.180059    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"]
Jul 07 16:48:52.182735 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.182470    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79c9458997-rrnwn"]
Jul 07 16:48:52.183793 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.182882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pjtgp"]
Jul 07 16:48:52.187114 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:52.186956    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7c2897_e46e_456b_b3d2_7f419224c6b3.slice/crio-dcc4247a9737d41ceca30c313dcbbe371e5a53eedaa10bc0bf0694fe1af90699 WatchSource:0}: Error finding container dcc4247a9737d41ceca30c313dcbbe371e5a53eedaa10bc0bf0694fe1af90699: Status 404 returned error can't find the container with id dcc4247a9737d41ceca30c313dcbbe371e5a53eedaa10bc0bf0694fe1af90699
Jul 07 16:48:52.188352 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:52.188300    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077f2741_19ab_4827_ace5_c11203efc785.slice/crio-7323b2dbc87f7420b94d2a1f1a49932c76f7d50f0c59f91848739d7a7642c310 WatchSource:0}: Error finding container 7323b2dbc87f7420b94d2a1f1a49932c76f7d50f0c59f91848739d7a7642c310: Status 404 returned error can't find the container with id 7323b2dbc87f7420b94d2a1f1a49932c76f7d50f0c59f91848739d7a7642c310
Jul 07 16:48:52.190343 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:48:52.189849    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f6c6b7_c8eb_4101_bdcc_bde59e056f86.slice/crio-1c90233627f5ea5e9a7e71b1185d49ec548c26e0b785746d59cfd026e4946241 WatchSource:0}: Error finding container 1c90233627f5ea5e9a7e71b1185d49ec548c26e0b785746d59cfd026e4946241: Status 404 returned error can't find the container with id 1c90233627f5ea5e9a7e71b1185d49ec548c26e0b785746d59cfd026e4946241
Jul 07 16:48:52.359215 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.359117    2559 patch_prober.go:28] interesting pod/image-registry-d8f87c964-kkjnq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 07 16:48:52.359215 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.359186    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d8f87c964-kkjnq" podUID="1321693f-e32d-4e48-9754-bceb0ec4431d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:48:52.553426 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.553236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:52.661340 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.661195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:52.680502 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.680405    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s562" event={"ID":"afc81d20-2e49-4284-96ed-866c03196411","Type":"ContainerStarted","Data":"e6b2faa554c1957e29b26f69cb0b4859c6fd3149ae887cecd587e013c310a22e"}
Jul 07 16:48:52.687601 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.687520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vp8v6" event={"ID":"fb7c2897-e46e-456b-b3d2-7f419224c6b3","Type":"ContainerStarted","Data":"dcc4247a9737d41ceca30c313dcbbe371e5a53eedaa10bc0bf0694fe1af90699"}
Jul 07 16:48:52.703106 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.703022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2" event={"ID":"332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e","Type":"ContainerStarted","Data":"655ffe33347929ef64226607f87af7f5ad1cef35a4d30922b2d9a4736e809636"}
Jul 07 16:48:52.710775 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.710701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c57bv" event={"ID":"d273870c-4ed6-44f4-be7c-669c1c013396","Type":"ContainerStarted","Data":"0d27544f5401bb44539828394d2c02b7015251d9b26de58905a4210bab2a0c10"}
Jul 07 16:48:52.715580 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.715488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn" event={"ID":"17f6c6b7-c8eb-4101-bdcc-bde59e056f86","Type":"ContainerStarted","Data":"1c90233627f5ea5e9a7e71b1185d49ec548c26e0b785746d59cfd026e4946241"}
Jul 07 16:48:52.719236 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.719095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w" event={"ID":"81b2546a-97bc-4af2-b109-5f05f40f8e23","Type":"ContainerStarted","Data":"bbdb38ab859150566c435fc2b88f7122d63663ac8de2f7482da05f7ce8c0180f"}
Jul 07 16:48:52.719236 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.719151    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w" event={"ID":"81b2546a-97bc-4af2-b109-5f05f40f8e23","Type":"ContainerStarted","Data":"bfc785f68d6cf893827d38a1e3952e1f322898530d3aa6901f0d00f74c57f603"}
Jul 07 16:48:52.719236 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.719179    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w" event={"ID":"81b2546a-97bc-4af2-b109-5f05f40f8e23","Type":"ContainerStarted","Data":"9c6bf35e4953eb9b51fbfc0c66c3aa9e5f23665b15b12de692da2b256432dbdb"}
Jul 07 16:48:52.722725 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.722637    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pjtgp" event={"ID":"077f2741-19ab-4827-ace5-c11203efc785","Type":"ContainerStarted","Data":"7323b2dbc87f7420b94d2a1f1a49932c76f7d50f0c59f91848739d7a7642c310"}
Jul 07 16:48:52.732820 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.732731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6" event={"ID":"eb0a2918-c0c2-4900-a335-ccb31870a05b","Type":"ContainerStarted","Data":"bc742470b93585c738f147ae2d7a88437dc775deb3182aba0f40249bc1976cc9"}
Jul 07 16:48:52.740907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.739837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-fjddn" event={"ID":"232694f9-b34b-49bd-b61a-e72e39479653","Type":"ContainerStarted","Data":"5483db6189790d425aed04102f0203e588c8abac05c08928ffe5b30a90c1ac1a"}
Jul 07 16:48:52.740907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.740869    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-fjddn"
Jul 07 16:48:52.751559 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.751536    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-fjddn"
Jul 07 16:48:52.785024 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:52.784937    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-fjddn" podStartSLOduration=1.6286598570000002 podStartE2EDuration="19.784908875s" totalImagesPullingTime="18.156249018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:52.762569683 +0000 UTC m=+71.026106316" watchObservedRunningTime="2026-07-07 16:48:52.784908875 +0000 UTC m=+71.048445509"
Jul 07 16:48:55.610000 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.609456    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d8f87c964-kkjnq"
Jul 07 16:48:55.630750 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.630599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:55.732044 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.732010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:55.828236 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.828192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:55.891583 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.889910    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f888d49cf-qndtm"]
Jul 07 16:48:55.911596 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.911509    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:55.917478 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.917452    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f888d49cf-qndtm"]
Jul 07 16:48:55.917615 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.917567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:55.921508 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.921486    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 16:48:55.921713 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.921690    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-54jkp\""
Jul 07 16:48:55.921794 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.921757    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 16:48:55.921915 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.921776    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 16:48:55.922852 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.922022    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 16:48:55.922852 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.921696    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 16:48:55.945963 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:55.945939    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 16:48:56.000544 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.000193    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-config\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.000544 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.000236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-service-ca\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.000544 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.000279    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-oauth-serving-cert\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.000544 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.000311    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-serving-cert\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.000544 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.000363    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-oauth-config\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.000544 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.000387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-trusted-ca-bundle\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.000544 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.000416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngpwd\" (UniqueName: \"kubernetes.io/projected/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-kube-api-access-ngpwd\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.101719 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.101667    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-oauth-config\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.101719 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.101716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-trusted-ca-bundle\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.102132 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.101746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngpwd\" (UniqueName: \"kubernetes.io/projected/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-kube-api-access-ngpwd\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.102132 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.101805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-config\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.102132 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.101832    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-service-ca\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.102132 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.101869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-oauth-serving-cert\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.102132 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.101894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-serving-cert\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.103269 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.103240    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-service-ca\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.103691 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.103622    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-config\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.103691 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.103629    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-oauth-serving-cert\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.104318 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.104289    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-trusted-ca-bundle\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.118144 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.110553    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-oauth-config\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.118144 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.116362    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngpwd\" (UniqueName: \"kubernetes.io/projected/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-kube-api-access-ngpwd\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.118606 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.118583    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-serving-cert\") pod \"console-6f888d49cf-qndtm\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") " pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.242763 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.242642    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:56.248705 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.248343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:48:56.310967 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.310908    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:56.599968 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.599902    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:56.704197 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:56.704142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:57.432904 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:57.432866    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:57.534610 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:57.534580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:58.047861 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.047828    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f888d49cf-qndtm"]
Jul 07 16:48:58.087481 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.087413    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d66b776bb-vw7cd"]
Jul 07 16:48:58.119028 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.118937    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d66b776bb-vw7cd"]
Jul 07 16:48:58.119189 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.119056    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.220023 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.219989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-oauth-config\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.220182 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.220042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-serving-cert\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.220182 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.220064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-oauth-serving-cert\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.220182 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.220115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cft6x\" (UniqueName: \"kubernetes.io/projected/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-kube-api-access-cft6x\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.220182 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.220159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-config\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.220385 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.220190    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-service-ca\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.220385 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.220221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-trusted-ca-bundle\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.321668 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.321581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-service-ca\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.321668 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.321636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-trusted-ca-bundle\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.321944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.321681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-oauth-config\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.321944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.321705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-serving-cert\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.321944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.321737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-oauth-serving-cert\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.321944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.321766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cft6x\" (UniqueName: \"kubernetes.io/projected/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-kube-api-access-cft6x\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.321944 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.321802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-config\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.323526 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.323393    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-config\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.324390 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.324343    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-service-ca\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.324573 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.324541    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-oauth-serving-cert\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.324878 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.324844    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-trusted-ca-bundle\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.325122 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.325102    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-serving-cert\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.326275 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.326253    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-oauth-config\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.338282 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.338263    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cft6x\" (UniqueName: \"kubernetes.io/projected/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-kube-api-access-cft6x\") pod \"console-5d66b776bb-vw7cd\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") " pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:58.432177 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:58.432141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:48:59.708169 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:59.708131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:59.814509 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:48:59.814430    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:00.837373 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:00.837300    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:00.938273 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:00.938227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:01.149187 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:01.149100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:01.249567 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:01.249539    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:01.751196 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:01.751167    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d66b776bb-vw7cd"]
Jul 07 16:49:01.757434 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:49:01.757402    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733c6324_8fb2_4d0a_a9e9_b6bd0fbf3772.slice/crio-082af5c9a67a45f5315bfa46bb131b3005c3cab8feb986407527bf40ac354570 WatchSource:0}: Error finding container 082af5c9a67a45f5315bfa46bb131b3005c3cab8feb986407527bf40ac354570: Status 404 returned error can't find the container with id 082af5c9a67a45f5315bfa46bb131b3005c3cab8feb986407527bf40ac354570
Jul 07 16:49:01.769351 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:01.769325    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f888d49cf-qndtm"]
Jul 07 16:49:01.774146 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:49:01.774085    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e1a63d_cc15_4efd_a5b2_ec82166a3bbb.slice/crio-8762f7e8c5434af28c30eeecdd2a6c0a4c4d01f42e3496d90b3091c8432a2732 WatchSource:0}: Error finding container 8762f7e8c5434af28c30eeecdd2a6c0a4c4d01f42e3496d90b3091c8432a2732: Status 404 returned error can't find the container with id 8762f7e8c5434af28c30eeecdd2a6c0a4c4d01f42e3496d90b3091c8432a2732
Jul 07 16:49:01.795433 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:01.795371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s562" event={"ID":"afc81d20-2e49-4284-96ed-866c03196411","Type":"ContainerStarted","Data":"0d0170e9ad4403eef2769de7c4b2996111e1aa5916654c06df3d5f32df22cff6"}
Jul 07 16:49:01.797493 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:01.797466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d66b776bb-vw7cd" event={"ID":"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772","Type":"ContainerStarted","Data":"082af5c9a67a45f5315bfa46bb131b3005c3cab8feb986407527bf40ac354570"}
Jul 07 16:49:01.800312 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:01.800287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w" event={"ID":"81b2546a-97bc-4af2-b109-5f05f40f8e23","Type":"ContainerStarted","Data":"fa75a04526031b8ce113087a1eb00b46f8669dcf889a7cff1fd6cfeee9032c5d"}
Jul 07 16:49:01.802786 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:01.802761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f888d49cf-qndtm" event={"ID":"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb","Type":"ContainerStarted","Data":"8762f7e8c5434af28c30eeecdd2a6c0a4c4d01f42e3496d90b3091c8432a2732"}
Jul 07 16:49:01.829777 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:01.829723    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-mkn4w" podStartSLOduration=7.294103718 podStartE2EDuration="16.829701881s" totalImagesPullingTime="9.535598163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:01.827425908 +0000 UTC m=+80.090962540" watchObservedRunningTime="2026-07-07 16:49:01.829701881 +0000 UTC m=+80.093238516"
Jul 07 16:49:02.814640 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.814561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn" event={"ID":"17f6c6b7-c8eb-4101-bdcc-bde59e056f86","Type":"ContainerStarted","Data":"56f1a4cb43bf20e7efa8046424586befa3a0dd1a8774102273a538be437c976c"}
Jul 07 16:49:02.815194 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.814655    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn" event={"ID":"17f6c6b7-c8eb-4101-bdcc-bde59e056f86","Type":"ContainerStarted","Data":"e50d02215b39fceb8ecee34e315087a77578663140cb466bee0823a1554bcd1f"}
Jul 07 16:49:02.815194 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.814707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn" event={"ID":"17f6c6b7-c8eb-4101-bdcc-bde59e056f86","Type":"ContainerStarted","Data":"57efc919b0a5518a8f2e991c3314a50e472e92194f12278ad8e7ad6a1e7c481f"}
Jul 07 16:49:02.816515 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.816481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pjtgp" event={"ID":"077f2741-19ab-4827-ace5-c11203efc785","Type":"ContainerStarted","Data":"7b226718a33f3f4b1d683a9f33977c818f55f6d0ee6f28106fd0f75e3493ce34"}
Jul 07 16:49:02.816648 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.816619    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:49:02.819357 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.819333    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6" event={"ID":"eb0a2918-c0c2-4900-a335-ccb31870a05b","Type":"ContainerStarted","Data":"a38228b73d6909043f13814fddebbd78486fb904cc0e0546f26c1c4887192bab"}
Jul 07 16:49:02.819454 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.819386    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6" event={"ID":"eb0a2918-c0c2-4900-a335-ccb31870a05b","Type":"ContainerStarted","Data":"1a552ea853bcde0900f531fdd5e5519451cecadba7d876ec26c6d867b3532f6b"}
Jul 07 16:49:02.819454 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.819414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6" event={"ID":"eb0a2918-c0c2-4900-a335-ccb31870a05b","Type":"ContainerStarted","Data":"180c6c0f4ce0fbf7119074b72d2c28c35c4caccc1a4298c1c09aa68ebe65548c"}
Jul 07 16:49:02.821731 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.821659    2559 generic.go:356] "Generic (PLEG): container finished" podID="afc81d20-2e49-4284-96ed-866c03196411" containerID="0d0170e9ad4403eef2769de7c4b2996111e1aa5916654c06df3d5f32df22cff6" exitCode=0
Jul 07 16:49:02.824179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.822321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s562" event={"ID":"afc81d20-2e49-4284-96ed-866c03196411","Type":"ContainerDied","Data":"0d0170e9ad4403eef2769de7c4b2996111e1aa5916654c06df3d5f32df22cff6"}
Jul 07 16:49:02.828040 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.827271    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vp8v6" event={"ID":"fb7c2897-e46e-456b-b3d2-7f419224c6b3","Type":"ContainerStarted","Data":"18026e366da0c0bde117f3f3c2974fd12d2e0206a263d78798fea01af0e3d912"}
Jul 07 16:49:02.828040 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.827336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vp8v6" event={"ID":"fb7c2897-e46e-456b-b3d2-7f419224c6b3","Type":"ContainerStarted","Data":"40ec813b642e5541377fcf38335a1db160a870871b76f48e2a47a2a9ea2bcd56"}
Jul 07 16:49:02.830674 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.830627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2" event={"ID":"332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e","Type":"ContainerStarted","Data":"b6928df4fc8885245b0f6be740263d6c4476c0b146e5cbd8cc9ba899e567c1a2"}
Jul 07 16:49:02.832865 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.831505    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pjtgp" podStartSLOduration=70.411910188 podStartE2EDuration="1m19.831488458s" totalImagesPullingTime="9.41957827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:02.830186363 +0000 UTC m=+81.093723023" watchObservedRunningTime="2026-07-07 16:49:02.831488458 +0000 UTC m=+81.095025093"
Jul 07 16:49:02.832865 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.831643    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:49:02.834204 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.834039    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c57bv" event={"ID":"d273870c-4ed6-44f4-be7c-669c1c013396","Type":"ContainerStarted","Data":"8e4722b17f06bf576f3575a4bcbdbaf6e4052b307ebf004ac63cafcacee5b7a6"}
Jul 07 16:49:02.837947 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.837929    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2"
Jul 07 16:49:02.869851 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.869804    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vp8v6" podStartSLOduration=70.450171326 podStartE2EDuration="1m19.869790096s" totalImagesPullingTime="9.41961877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:02.847889742 +0000 UTC m=+81.111426375" watchObservedRunningTime="2026-07-07 16:49:02.869790096 +0000 UTC m=+81.133326722"
Jul 07 16:49:02.890183 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.889834    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gfdb6" podStartSLOduration=8.236516322 podStartE2EDuration="17.889815051s" totalImagesPullingTime="9.653298729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:02.888473757 +0000 UTC m=+81.152010425" watchObservedRunningTime="2026-07-07 16:49:02.889815051 +0000 UTC m=+81.153351684"
Jul 07 16:49:02.903918 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.903878    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6nxx2" podStartSLOduration=3.478978666 podStartE2EDuration="12.903863735s" totalImagesPullingTime="9.424885069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:02.903040108 +0000 UTC m=+81.166576740" watchObservedRunningTime="2026-07-07 16:49:02.903863735 +0000 UTC m=+81.167400367"
Jul 07 16:49:02.923113 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.923060    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c57bv" podStartSLOduration=70.486544481 podStartE2EDuration="1m19.923040687s" totalImagesPullingTime="9.436496206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:02.921756723 +0000 UTC m=+81.185293382" watchObservedRunningTime="2026-07-07 16:49:02.923040687 +0000 UTC m=+81.186577321"
Jul 07 16:49:02.940155 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:02.940126    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:03.037506 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:03.037471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:03.137365 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:03.137319    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:03.840596 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:03.840567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s562" event={"ID":"afc81d20-2e49-4284-96ed-866c03196411","Type":"ContainerStarted","Data":"946150e97b5ccc8b7f528dcb833de2e095ac5786a5195c2d2084ad4e977f4e34"}
Jul 07 16:49:03.841083 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:03.840624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s562" event={"ID":"afc81d20-2e49-4284-96ed-866c03196411","Type":"ContainerStarted","Data":"bebd173f592d72eb9cde340b965f39771ffe945116ea20b7e4671977e7d4ab78"}
Jul 07 16:49:03.843777 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:03.843749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn" event={"ID":"17f6c6b7-c8eb-4101-bdcc-bde59e056f86","Type":"ContainerStarted","Data":"a426981aa26dbd024f51163091bdb6a9969142dabc8b63086acd4aece645d43f"}
Jul 07 16:49:03.843936 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:03.843819    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn" event={"ID":"17f6c6b7-c8eb-4101-bdcc-bde59e056f86","Type":"ContainerStarted","Data":"be7176978fef498d8c3ee895791cea326dd486b57043f721e6a0a402fa86faaa"}
Jul 07 16:49:03.843936 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:03.843860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn" event={"ID":"17f6c6b7-c8eb-4101-bdcc-bde59e056f86","Type":"ContainerStarted","Data":"f3a0767679a363d6b09563deb906ed2ac4811c6b5ad6e13dc3c3d4637cd97ae8"}
Jul 07 16:49:03.863442 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:03.863310    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8s562" podStartSLOduration=8.918224742 podStartE2EDuration="18.863286853s" totalImagesPullingTime="9.781602926s" totalInitContainerRuntime="163.459185ms" isStatefulPod=true podCreationTimestamp="2026-07-07 16:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:03.860249794 +0000 UTC m=+82.123786464" watchObservedRunningTime="2026-07-07 16:49:03.863286853 +0000 UTC m=+82.126823486"
Jul 07 16:49:03.886173 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:03.886111    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn" podStartSLOduration=6.279999339 podStartE2EDuration="16.886088129s" totalImagesPullingTime="10.60608879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:03.884632813 +0000 UTC m=+82.148169474" watchObservedRunningTime="2026-07-07 16:49:03.886088129 +0000 UTC m=+82.149624764"
Jul 07 16:49:04.251759 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:04.251671    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:04.353039 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:04.352967    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:04.850910 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:04.850790    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:49:05.598815 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:05.598775    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:05.853876 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:05.853844    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d66b776bb-vw7cd" event={"ID":"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772","Type":"ContainerStarted","Data":"bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956"}
Jul 07 16:49:05.855465 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:05.855439    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f888d49cf-qndtm" event={"ID":"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb","Type":"ContainerStarted","Data":"c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b"}
Jul 07 16:49:05.873689 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:05.873649    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d66b776bb-vw7cd" podStartSLOduration=4.101748088 podStartE2EDuration="7.873633623s" totalImagesPullingTime="3.771885535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:05.872109704 +0000 UTC m=+84.135646347" watchObservedRunningTime="2026-07-07 16:49:05.873633623 +0000 UTC m=+84.137170292"
Jul 07 16:49:05.891747 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:05.891660    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f888d49cf-qndtm" podStartSLOduration=7.141868238 podStartE2EDuration="10.891644133s" totalImagesPullingTime="3.749775895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:05.891249626 +0000 UTC m=+84.154786281" watchObservedRunningTime="2026-07-07 16:49:05.891644133 +0000 UTC m=+84.155180767"
Jul 07 16:49:06.249475 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:06.249400    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:49:08.433334 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:08.433299    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:49:08.433715 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:08.433374    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:49:08.438098 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:08.438078    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:49:08.806827 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:08.806775    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:08.862934 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:08.862898    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:08.868870 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:08.868847    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:49:09.027070 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:09.027006    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:09.134658 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:09.134574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:09.188761 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:09.188724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:10.860838 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:10.860807    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79c9458997-rrnwn"
Jul 07 16:49:15.787259 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:15.787221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:15.890690 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:15.890638    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:25.591259 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:25.591216    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:25.691928 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:25.691894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:30.875631 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:30.875571    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6f888d49cf-qndtm" podUID="b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" containerName="console" containerID="cri-o://c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b" gracePeriod=15
Jul 07 16:49:31.168638 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.168581    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:49:31.296104 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296068    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-service-ca\") pod \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") "
Jul 07 16:49:31.296283 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296113    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-config\") pod \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") "
Jul 07 16:49:31.296283 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296148    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-oauth-serving-cert\") pod \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") "
Jul 07 16:49:31.296283 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296180    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-trusted-ca-bundle\") pod \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") "
Jul 07 16:49:31.296283 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296212    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-oauth-config\") pod \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") "
Jul 07 16:49:31.296283 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296251    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-serving-cert\") pod \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") "
Jul 07 16:49:31.296520 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296297    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngpwd\" (UniqueName: \"kubernetes.io/projected/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-kube-api-access-ngpwd\") pod \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\" (UID: \"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb\") "
Jul 07 16:49:31.296557 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296520    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-config" (OuterVolumeSpecName: "console-config") pod "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" (UID: "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:31.296557 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296536    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" (UID: "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:31.296631 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296585    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" (UID: "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:31.296815 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.296781    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-service-ca" (OuterVolumeSpecName: "service-ca") pod "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" (UID: "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:31.298616 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.298584    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-kube-api-access-ngpwd" (OuterVolumeSpecName: "kube-api-access-ngpwd") pod "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" (UID: "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb"). InnerVolumeSpecName "kube-api-access-ngpwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:49:31.298707 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.298588    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" (UID: "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:31.298707 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.298658    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" (UID: "b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:31.397663 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.397638    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngpwd\" (UniqueName: \"kubernetes.io/projected/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-kube-api-access-ngpwd\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:49:31.397663 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.397662    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-service-ca\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:49:31.397801 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.397671    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:49:31.397801 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.397681    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-oauth-serving-cert\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:49:31.397801 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.397689    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-trusted-ca-bundle\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:49:31.397801 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.397698    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-oauth-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:49:31.397801 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.397708    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb-console-serving-cert\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:49:31.949764 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.949725    2559 generic.go:356] "Generic (PLEG): container finished" podID="b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" containerID="c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b" exitCode=0
Jul 07 16:49:31.950164 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.949771    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f888d49cf-qndtm"
Jul 07 16:49:31.950164 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.949803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f888d49cf-qndtm" event={"ID":"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb","Type":"ContainerDied","Data":"c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b"}
Jul 07 16:49:31.950164 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.949852    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f888d49cf-qndtm" event={"ID":"b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb","Type":"ContainerDied","Data":"8762f7e8c5434af28c30eeecdd2a6c0a4c4d01f42e3496d90b3091c8432a2732"}
Jul 07 16:49:31.950164 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.949871    2559 scope.go:117] "RemoveContainer" containerID="c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b"
Jul 07 16:49:31.966360 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.966334    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f888d49cf-qndtm"]
Jul 07 16:49:31.970706 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.970684    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f888d49cf-qndtm"]
Jul 07 16:49:31.971658 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.971643    2559 scope.go:117] "RemoveContainer" containerID="c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b"
Jul 07 16:49:31.971956 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:49:31.971930    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b\": container with ID starting with c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b not found: ID does not exist" containerID="c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b"
Jul 07 16:49:31.972062 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:31.971958    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b"} err="failed to get container status \"c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b\": rpc error: code = NotFound desc = could not find container \"c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b\": container with ID starting with c36c5dd66e571260a5de847ee19185ae5cf23bf03e0066d672259e2c85a4447b not found: ID does not exist"
Jul 07 16:49:33.334822 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:33.334787    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" path="/var/lib/kubelet/pods/b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb/volumes"
Jul 07 16:49:33.846448 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:33.846412    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pjtgp"
Jul 07 16:49:57.021817 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:49:57.021785    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:01.367698 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:01.367664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:01.428159 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:01.428127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:06.694504 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.694472    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dd7479777-g85ns"]
Jul 07 16:50:06.694966 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.694725    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" containerName="console"
Jul 07 16:50:06.694966 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.694735    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" containerName="console"
Jul 07 16:50:06.694966 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.694794    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e1a63d-cc15-4efd-a5b2-ec82166a3bbb" containerName="console"
Jul 07 16:50:06.697038 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.697021    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.712276 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.712250    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd7479777-g85ns"]
Jul 07 16:50:06.740593 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.740566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-oauth-serving-cert\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.740713 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.740604    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crv6p\" (UniqueName: \"kubernetes.io/projected/5546ee6a-e789-4744-a278-c78c4a7b71bb-kube-api-access-crv6p\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.740713 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.740622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-oauth-config\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.740713 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.740649    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-trusted-ca-bundle\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.740877 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.740716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-serving-cert\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.740877 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.740775    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-service-ca\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.740877 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.740822    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-config\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.752783 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.752760    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:06.841517 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.841490    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-service-ca\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.841631 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.841522    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-config\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.841631 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.841543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-oauth-serving-cert\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.841631 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.841576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crv6p\" (UniqueName: \"kubernetes.io/projected/5546ee6a-e789-4744-a278-c78c4a7b71bb-kube-api-access-crv6p\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.841631 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.841605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-oauth-config\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.841873 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.841636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-trusted-ca-bundle\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.841873 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.841668    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-serving-cert\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.842289 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.842264    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-service-ca\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.842386 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.842342    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-config\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.842386 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.842351    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-oauth-serving-cert\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.842715 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.842696    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-trusted-ca-bundle\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.844505 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.844478    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-oauth-config\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.844692 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.844672    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-serving-cert\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.849584 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.849549    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crv6p\" (UniqueName: \"kubernetes.io/projected/5546ee6a-e789-4744-a278-c78c4a7b71bb-kube-api-access-crv6p\") pod \"console-7dd7479777-g85ns\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") " pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:06.853411 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.853372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:06.954508 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:06.954421    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:07.006257 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:07.006232    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:07.133339 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:07.133300    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd7479777-g85ns"]
Jul 07 16:50:07.139870 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:50:07.139842    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5546ee6a_e789_4744_a278_c78c4a7b71bb.slice/crio-fe5aeb5addd233e8154235cc1d0ee94d1d1512cf9b15be4c7afba6950be6b36a WatchSource:0}: Error finding container fe5aeb5addd233e8154235cc1d0ee94d1d1512cf9b15be4c7afba6950be6b36a: Status 404 returned error can't find the container with id fe5aeb5addd233e8154235cc1d0ee94d1d1512cf9b15be4c7afba6950be6b36a
Jul 07 16:50:07.546363 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:07.546330    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:07.643909 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:07.643877    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:08.076797 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:08.076763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd7479777-g85ns" event={"ID":"5546ee6a-e789-4744-a278-c78c4a7b71bb","Type":"ContainerStarted","Data":"4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968"}
Jul 07 16:50:08.077233 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:08.076810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd7479777-g85ns" event={"ID":"5546ee6a-e789-4744-a278-c78c4a7b71bb","Type":"ContainerStarted","Data":"fe5aeb5addd233e8154235cc1d0ee94d1d1512cf9b15be4c7afba6950be6b36a"}
Jul 07 16:50:08.094245 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:08.094198    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dd7479777-g85ns" podStartSLOduration=2.094180124 podStartE2EDuration="2.094180124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:50:08.093179808 +0000 UTC m=+146.356716443" watchObservedRunningTime="2026-07-07 16:50:08.094180124 +0000 UTC m=+146.357716757"
Jul 07 16:50:11.604407 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:11.604368    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:13.300686 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:13.300643    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:13.361011 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:13.360960    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:15.786568 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:15.786533    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:15.842658 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:15.842625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:16.513237 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:16.513199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:17.007150 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:17.007125    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:17.007506 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:17.007168    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:17.011836 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:17.011816    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:17.112357 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:17.112330    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:50:17.161809 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:17.161784    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d66b776bb-vw7cd"]
Jul 07 16:50:17.269242 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:17.269161    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:17.372081 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:17.372047    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:30.806630 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:30.806592    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:30.903027 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:30.902994    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:42.181510 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.181464    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5d66b776bb-vw7cd" podUID="733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" containerName="console" containerID="cri-o://bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956" gracePeriod=15
Jul 07 16:50:42.426689 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.426667    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:50:42.520003 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.519958    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-config\") pod \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") "
Jul 07 16:50:42.520130 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520028    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-service-ca\") pod \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") "
Jul 07 16:50:42.520130 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520060    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-serving-cert\") pod \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") "
Jul 07 16:50:42.520130 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520078    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cft6x\" (UniqueName: \"kubernetes.io/projected/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-kube-api-access-cft6x\") pod \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") "
Jul 07 16:50:42.520130 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520121    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-oauth-config\") pod \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") "
Jul 07 16:50:42.520337 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520266    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-oauth-serving-cert\") pod \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") "
Jul 07 16:50:42.520337 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-trusted-ca-bundle\") pod \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\" (UID: \"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772\") "
Jul 07 16:50:42.520434 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520393    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-config" (OuterVolumeSpecName: "console-config") pod "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" (UID: "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:42.520562 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520538    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-service-ca" (OuterVolumeSpecName: "service-ca") pod "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" (UID: "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:42.520638 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520619    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:50:42.520638 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520604    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" (UID: "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:42.520757 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.520735    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" (UID: "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:50:42.522321 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.522285    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" (UID: "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:42.522439 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.522339    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" (UID: "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:50:42.522439 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.522337    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-kube-api-access-cft6x" (OuterVolumeSpecName: "kube-api-access-cft6x") pod "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" (UID: "733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772"). InnerVolumeSpecName "kube-api-access-cft6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:50:42.620916 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.620879    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-oauth-serving-cert\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:50:42.620916 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.620913    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-trusted-ca-bundle\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:50:42.620916 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.620924    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-service-ca\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:50:42.621091 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.620932    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-serving-cert\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:50:42.621091 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.620941    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cft6x\" (UniqueName: \"kubernetes.io/projected/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-kube-api-access-cft6x\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:50:42.621091 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:42.620951    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772-console-oauth-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:50:43.199053 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.199019    2559 generic.go:356] "Generic (PLEG): container finished" podID="733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" containerID="bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956" exitCode=0
Jul 07 16:50:43.199478 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.199060    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d66b776bb-vw7cd"
Jul 07 16:50:43.199478 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.199094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d66b776bb-vw7cd" event={"ID":"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772","Type":"ContainerDied","Data":"bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956"}
Jul 07 16:50:43.199478 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.199175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d66b776bb-vw7cd" event={"ID":"733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772","Type":"ContainerDied","Data":"082af5c9a67a45f5315bfa46bb131b3005c3cab8feb986407527bf40ac354570"}
Jul 07 16:50:43.199478 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.199209    2559 scope.go:117] "RemoveContainer" containerID="bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956"
Jul 07 16:50:43.207187 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.207171    2559 scope.go:117] "RemoveContainer" containerID="bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956"
Jul 07 16:50:43.207426 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:50:43.207408    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956\": container with ID starting with bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956 not found: ID does not exist" containerID="bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956"
Jul 07 16:50:43.207469 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.207434    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956"} err="failed to get container status \"bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956\": rpc error: code = NotFound desc = could not find container \"bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956\": container with ID starting with bcb858a086e0d9030f97c7dbf29224c136467fbcccf8b36a53d3acd131b69956 not found: ID does not exist"
Jul 07 16:50:43.218103 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.218078    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d66b776bb-vw7cd"]
Jul 07 16:50:43.221200 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.221176    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d66b776bb-vw7cd"]
Jul 07 16:50:43.336399 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:43.336374    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" path="/var/lib/kubelet/pods/733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772/volumes"
Jul 07 16:50:51.006476 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:50:51.006440    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:22.309152 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:22.309060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:22.408688 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:22.408655    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:22.733186 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:22.733102    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:22.840386 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:22.840347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:22.935507 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:22.935475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:23.733779 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.733748    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"]
Jul 07 16:51:23.734153 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.734036    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" containerName="console"
Jul 07 16:51:23.734153 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.734047    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" containerName="console"
Jul 07 16:51:23.734153 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.734089    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="733c6324-8fb2-4d0a-a9e9-b6bd0fbf3772" containerName="console"
Jul 07 16:51:23.735873 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.735858    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:23.738241 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.738219    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-58fml\""
Jul 07 16:51:23.738381 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.738264    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 07 16:51:23.738946 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.738930    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 07 16:51:23.745492 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.745471    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"]
Jul 07 16:51:23.801812 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.801785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:23.801942 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.801831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhx8\" (UniqueName: \"kubernetes.io/projected/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-kube-api-access-gqhx8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:23.801942 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.801860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:23.902858 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.902825    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:23.903032 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.902876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhx8\" (UniqueName: \"kubernetes.io/projected/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-kube-api-access-gqhx8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:23.903032 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.902906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:23.903234 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.903214    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:23.903277 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.903261    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:23.910730 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:23.910711    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhx8\" (UniqueName: \"kubernetes.io/projected/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-kube-api-access-gqhx8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:24.045035 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:24.045010    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:24.164772 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:24.164744    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"]
Jul 07 16:51:24.336622 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:24.336558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9" event={"ID":"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b","Type":"ContainerStarted","Data":"b8d30a86bd88212ef9bebd3f2068942355b68ce23bbe76c3b1302652fb2d75b3"}
Jul 07 16:51:31.364108 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:31.364040    2559 generic.go:356] "Generic (PLEG): container finished" podID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerID="7d8f9daa4ff07cd3f93609c13f811e38948bd8d348d3eb97e89f5ef6566a626c" exitCode=0
Jul 07 16:51:31.364108 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:31.364076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9" event={"ID":"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b","Type":"ContainerDied","Data":"7d8f9daa4ff07cd3f93609c13f811e38948bd8d348d3eb97e89f5ef6566a626c"}
Jul 07 16:51:33.738062 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:33.738033    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b4a6ad_2961_4a34_a63f_8f2bc5b24d1b.slice/crio-conmon-8e4e0b8df63afea3b2d145fe99b1e8ba7c5c74bd2388234123efdb87dc58c641.scope\": RecentStats: unable to find data in memory cache]"
Jul 07 16:51:34.378078 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:34.378047    2559 generic.go:356] "Generic (PLEG): container finished" podID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerID="8e4e0b8df63afea3b2d145fe99b1e8ba7c5c74bd2388234123efdb87dc58c641" exitCode=0
Jul 07 16:51:34.378252 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:34.378096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9" event={"ID":"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b","Type":"ContainerDied","Data":"8e4e0b8df63afea3b2d145fe99b1e8ba7c5c74bd2388234123efdb87dc58c641"}
Jul 07 16:51:40.407090 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:40.407024    2559 generic.go:356] "Generic (PLEG): container finished" podID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerID="d9833a8b561b47cd50664f535b91b9f5455305bc1997086fc4ffcb88243097c7" exitCode=0
Jul 07 16:51:40.407090 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:40.407071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9" event={"ID":"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b","Type":"ContainerDied","Data":"d9833a8b561b47cd50664f535b91b9f5455305bc1997086fc4ffcb88243097c7"}
Jul 07 16:51:41.530555 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.530533    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:41.650547 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.650513    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-bundle\") pod \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") "
Jul 07 16:51:41.650763 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.650587    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqhx8\" (UniqueName: \"kubernetes.io/projected/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-kube-api-access-gqhx8\") pod \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") "
Jul 07 16:51:41.650763 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.650619    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-util\") pod \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\" (UID: \"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b\") "
Jul 07 16:51:41.651251 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.651224    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-bundle" (OuterVolumeSpecName: "bundle") pod "35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" (UID: "35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:51:41.652840 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.652812    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-kube-api-access-gqhx8" (OuterVolumeSpecName: "kube-api-access-gqhx8") pod "35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" (UID: "35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b"). InnerVolumeSpecName "kube-api-access-gqhx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:51:41.654948 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.654930    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-util" (OuterVolumeSpecName: "util") pod "35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" (UID: "35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:51:41.751743 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.751687    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-bundle\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:51:41.751743 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.751708    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqhx8\" (UniqueName: \"kubernetes.io/projected/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-kube-api-access-gqhx8\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:51:41.751743 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:41.751716    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b-util\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:51:42.415816 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:42.415741    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9" event={"ID":"35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b","Type":"ContainerDied","Data":"b8d30a86bd88212ef9bebd3f2068942355b68ce23bbe76c3b1302652fb2d75b3"}
Jul 07 16:51:42.415816 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:42.415773    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589wgxt9"
Jul 07 16:51:42.415816 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:42.415785    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8d30a86bd88212ef9bebd3f2068942355b68ce23bbe76c3b1302652fb2d75b3"
Jul 07 16:51:44.709541 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:44.709506    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:44.807571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:44.807521    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:46.341456 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.341422    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"]
Jul 07 16:51:46.341905 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.341679    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerName="pull"
Jul 07 16:51:46.341905 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.341689    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerName="pull"
Jul 07 16:51:46.341905 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.341697    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerName="extract"
Jul 07 16:51:46.341905 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.341703    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerName="extract"
Jul 07 16:51:46.341905 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.341717    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerName="util"
Jul 07 16:51:46.341905 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.341722    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerName="util"
Jul 07 16:51:46.341905 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.341767    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b4a6ad-2961-4a34-a63f-8f2bc5b24d1b" containerName="extract"
Jul 07 16:51:46.345814 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.345798    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:51:46.349451 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.349431    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 07 16:51:46.349571 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.349486    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-94bml\""
Jul 07 16:51:46.349824 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.349810    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 07 16:51:46.349878 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.349851    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 07 16:51:46.355487 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.355467    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"]
Jul 07 16:51:46.480536 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.480511    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgt27\" (UniqueName: \"kubernetes.io/projected/67d74c97-4594-436c-b3bc-ad70c4fda673-kube-api-access-rgt27\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl\" (UID: \"67d74c97-4594-436c-b3bc-ad70c4fda673\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:51:46.480700 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.480539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/67d74c97-4594-436c-b3bc-ad70c4fda673-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl\" (UID: \"67d74c97-4594-436c-b3bc-ad70c4fda673\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:51:46.581354 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.581325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgt27\" (UniqueName: \"kubernetes.io/projected/67d74c97-4594-436c-b3bc-ad70c4fda673-kube-api-access-rgt27\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl\" (UID: \"67d74c97-4594-436c-b3bc-ad70c4fda673\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:51:46.581354 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.581354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/67d74c97-4594-436c-b3bc-ad70c4fda673-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl\" (UID: \"67d74c97-4594-436c-b3bc-ad70c4fda673\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:51:46.583524 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.583499    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/67d74c97-4594-436c-b3bc-ad70c4fda673-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl\" (UID: \"67d74c97-4594-436c-b3bc-ad70c4fda673\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:51:46.589518 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.589487    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgt27\" (UniqueName: \"kubernetes.io/projected/67d74c97-4594-436c-b3bc-ad70c4fda673-kube-api-access-rgt27\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl\" (UID: \"67d74c97-4594-436c-b3bc-ad70c4fda673\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:51:46.655902 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.655843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:51:46.780628 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:46.780595    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"]
Jul 07 16:51:46.783485 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:51:46.783455    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d74c97_4594_436c_b3bc_ad70c4fda673.slice/crio-bbf025e9e9978579424ce63bc6e38b3c46219dee2d779e811750742af7f25520 WatchSource:0}: Error finding container bbf025e9e9978579424ce63bc6e38b3c46219dee2d779e811750742af7f25520: Status 404 returned error can't find the container with id bbf025e9e9978579424ce63bc6e38b3c46219dee2d779e811750742af7f25520
Jul 07 16:51:47.434363 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:47.434334    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl" event={"ID":"67d74c97-4594-436c-b3bc-ad70c4fda673","Type":"ContainerStarted","Data":"bbf025e9e9978579424ce63bc6e38b3c46219dee2d779e811750742af7f25520"}
Jul 07 16:51:50.448033 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.447932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl" event={"ID":"67d74c97-4594-436c-b3bc-ad70c4fda673","Type":"ContainerStarted","Data":"a5be5314df32e5db2b08ed3f500e57863c7dda28b2c935453301005c32ea11d4"}
Jul 07 16:51:50.448362 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.448062    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:51:50.470063 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.470016    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl" podStartSLOduration=1.20191106 podStartE2EDuration="4.470003229s" totalImagesPullingTime="3.268092169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:51:50.468644228 +0000 UTC m=+248.732180880" watchObservedRunningTime="2026-07-07 16:51:50.470003229 +0000 UTC m=+248.733539860"
Jul 07 16:51:50.491056 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.491010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:50.753770 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.753746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:50.761144 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.761120    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-hvzwl"]
Jul 07 16:51:50.764248 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.764232    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:50.767179 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.767159    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 07 16:51:50.767294 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.767184    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-792fl\""
Jul 07 16:51:50.767339 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.767299    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 07 16:51:50.773133 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.773098    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-hvzwl"]
Jul 07 16:51:50.916340 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.916316    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczch\" (UniqueName: \"kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-kube-api-access-wczch\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:50.916476 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.916351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7de013cf-4ea5-40a0-b137-4e40ee0db731-cabundle0\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:50.916476 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.916382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-certificates\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:50.976092 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.976067    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"]
Jul 07 16:51:50.979300 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.979282    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:50.980267 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.980240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:50.981398 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.981380    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 07 16:51:50.989421 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:50.989401    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"]
Jul 07 16:51:51.016893 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.016836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczch\" (UniqueName: \"kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-kube-api-access-wczch\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:51.016893 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.016869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7de013cf-4ea5-40a0-b137-4e40ee0db731-cabundle0\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:51.016893 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.016889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-certificates\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:51.017118 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.016991    2559 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 07 16:51:51.017118 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.017005    2559 secret.go:281] references non-existent secret key: ca.crt
Jul 07 16:51:51.017118 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.017013    2559 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 07 16:51:51.017118 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.017023    2559 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-hvzwl: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 07 16:51:51.017118 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.017074    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-certificates podName:7de013cf-4ea5-40a0-b137-4e40ee0db731 nodeName:}" failed. No retries permitted until 2026-07-07 16:51:51.517056293 +0000 UTC m=+249.780592902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-certificates") pod "keda-operator-68bff59c-hvzwl" (UID: "7de013cf-4ea5-40a0-b137-4e40ee0db731") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 07 16:51:51.017553 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.017531    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7de013cf-4ea5-40a0-b137-4e40ee0db731-cabundle0\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:51.027277 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.027258    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczch\" (UniqueName: \"kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-kube-api-access-wczch\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:51.079584 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.079564    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:51.117658 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.117637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xbq\" (UniqueName: \"kubernetes.io/projected/b5bc9a78-fefc-42a5-9006-43665c1d1494-kube-api-access-k2xbq\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.117764 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.117677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b5bc9a78-fefc-42a5-9006-43665c1d1494-certificates\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.117764 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.117704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b5bc9a78-fefc-42a5-9006-43665c1d1494-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.218412 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.218386    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b5bc9a78-fefc-42a5-9006-43665c1d1494-certificates\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.218525 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.218419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b5bc9a78-fefc-42a5-9006-43665c1d1494-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.218525 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.218478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xbq\" (UniqueName: \"kubernetes.io/projected/b5bc9a78-fefc-42a5-9006-43665c1d1494-kube-api-access-k2xbq\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.218639 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.218521    2559 secret.go:281] references non-existent secret key: tls.crt
Jul 07 16:51:51.218639 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.218540    2559 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 07 16:51:51.218639 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.218559    2559 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 07 16:51:51.218639 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.218576    2559 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-tffqm: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 07 16:51:51.218639 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.218630    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5bc9a78-fefc-42a5-9006-43665c1d1494-certificates podName:b5bc9a78-fefc-42a5-9006-43665c1d1494 nodeName:}" failed. No retries permitted until 2026-07-07 16:51:51.718607918 +0000 UTC m=+249.982144529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b5bc9a78-fefc-42a5-9006-43665c1d1494-certificates") pod "keda-metrics-apiserver-559f75f947-tffqm" (UID: "b5bc9a78-fefc-42a5-9006-43665c1d1494") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 07 16:51:51.218875 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.218851    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b5bc9a78-fefc-42a5-9006-43665c1d1494-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.227503 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.227478    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xbq\" (UniqueName: \"kubernetes.io/projected/b5bc9a78-fefc-42a5-9006-43665c1d1494-kube-api-access-k2xbq\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.292215 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.292151    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:51.520816 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.520785    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-certificates\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:51.521285 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.520963    2559 secret.go:281] references non-existent secret key: ca.crt
Jul 07 16:51:51.521285 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.520996    2559 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 07 16:51:51.521285 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.521006    2559 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-hvzwl: references non-existent secret key: ca.crt
Jul 07 16:51:51.521285 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:51:51.521063    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-certificates podName:7de013cf-4ea5-40a0-b137-4e40ee0db731 nodeName:}" failed. No retries permitted until 2026-07-07 16:51:52.521043675 +0000 UTC m=+250.784580298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-certificates") pod "keda-operator-68bff59c-hvzwl" (UID: "7de013cf-4ea5-40a0-b137-4e40ee0db731") : references non-existent secret key: ca.crt
Jul 07 16:51:51.584066 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.583967    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:51.590500 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.590477    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-vlvfj"]
Jul 07 16:51:51.593839 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.593825    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:51:51.596112 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.596093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 07 16:51:51.604016 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.603998    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-vlvfj"]
Jul 07 16:51:51.682968 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.682935    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:51.722527 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.722498    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b5bc9a78-fefc-42a5-9006-43665c1d1494-certificates\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.722640 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.722539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wbj\" (UniqueName: \"kubernetes.io/projected/f3ffcd66-94b9-4f10-b6e2-82f551fe6b65-kube-api-access-79wbj\") pod \"keda-admission-5fbd5c4644-vlvfj\" (UID: \"f3ffcd66-94b9-4f10-b6e2-82f551fe6b65\") " pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:51:51.722723 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.722703    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3ffcd66-94b9-4f10-b6e2-82f551fe6b65-certificates\") pod \"keda-admission-5fbd5c4644-vlvfj\" (UID: \"f3ffcd66-94b9-4f10-b6e2-82f551fe6b65\") " pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:51:51.724963 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.724945    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b5bc9a78-fefc-42a5-9006-43665c1d1494-certificates\") pod \"keda-metrics-apiserver-559f75f947-tffqm\" (UID: \"b5bc9a78-fefc-42a5-9006-43665c1d1494\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.823780 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.823755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3ffcd66-94b9-4f10-b6e2-82f551fe6b65-certificates\") pod \"keda-admission-5fbd5c4644-vlvfj\" (UID: \"f3ffcd66-94b9-4f10-b6e2-82f551fe6b65\") " pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:51:51.823892 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.823808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wbj\" (UniqueName: \"kubernetes.io/projected/f3ffcd66-94b9-4f10-b6e2-82f551fe6b65-kube-api-access-79wbj\") pod \"keda-admission-5fbd5c4644-vlvfj\" (UID: \"f3ffcd66-94b9-4f10-b6e2-82f551fe6b65\") " pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:51:51.826226 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.826201    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f3ffcd66-94b9-4f10-b6e2-82f551fe6b65-certificates\") pod \"keda-admission-5fbd5c4644-vlvfj\" (UID: \"f3ffcd66-94b9-4f10-b6e2-82f551fe6b65\") " pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:51:51.832219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.832196    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wbj\" (UniqueName: \"kubernetes.io/projected/f3ffcd66-94b9-4f10-b6e2-82f551fe6b65-kube-api-access-79wbj\") pod \"keda-admission-5fbd5c4644-vlvfj\" (UID: \"f3ffcd66-94b9-4f10-b6e2-82f551fe6b65\") " pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:51:51.885127 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.885059    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:51.888866 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.888848    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:51.903868 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.903848    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:51:51.987413 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:51.987341    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:52.024776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.024748    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"]
Jul 07 16:51:52.026193 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:51:52.026170    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bc9a78_fefc_42a5_9006_43665c1d1494.slice/crio-24c6423721b95fed067c8335b47ce42d83dece15d24f08cce06d10d30e3aec94 WatchSource:0}: Error finding container 24c6423721b95fed067c8335b47ce42d83dece15d24f08cce06d10d30e3aec94: Status 404 returned error can't find the container with id 24c6423721b95fed067c8335b47ce42d83dece15d24f08cce06d10d30e3aec94
Jul 07 16:51:52.042177 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.042155    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-vlvfj"]
Jul 07 16:51:52.044561 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:51:52.044539    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ffcd66_94b9_4f10_b6e2_82f551fe6b65.slice/crio-9ef45cc2e0561e25fed0b328910602fa466b83dd895e5b7c300261f2c230d727 WatchSource:0}: Error finding container 9ef45cc2e0561e25fed0b328910602fa466b83dd895e5b7c300261f2c230d727: Status 404 returned error can't find the container with id 9ef45cc2e0561e25fed0b328910602fa466b83dd895e5b7c300261f2c230d727
Jul 07 16:51:52.085220 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.085192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:52.287105 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.287074    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:52.343656 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.343625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:52.456506 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.456470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj" event={"ID":"f3ffcd66-94b9-4f10-b6e2-82f551fe6b65","Type":"ContainerStarted","Data":"9ef45cc2e0561e25fed0b328910602fa466b83dd895e5b7c300261f2c230d727"}
Jul 07 16:51:52.457493 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.457470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm" event={"ID":"b5bc9a78-fefc-42a5-9006-43665c1d1494","Type":"ContainerStarted","Data":"24c6423721b95fed067c8335b47ce42d83dece15d24f08cce06d10d30e3aec94"}
Jul 07 16:51:52.530438 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.530408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-certificates\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:52.532690 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.532671    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7de013cf-4ea5-40a0-b137-4e40ee0db731-certificates\") pod \"keda-operator-68bff59c-hvzwl\" (UID: \"7de013cf-4ea5-40a0-b137-4e40ee0db731\") " pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:52.574391 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.574341    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:52.697748 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:52.697720    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-hvzwl"]
Jul 07 16:51:52.700859 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:51:52.700836    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de013cf_4ea5_40a0_b137_4e40ee0db731.slice/crio-c192190506ed34f4f5fbd11852dd57f0198f0d4f0f56cc95e7be4fc3493d5a98 WatchSource:0}: Error finding container c192190506ed34f4f5fbd11852dd57f0198f0d4f0f56cc95e7be4fc3493d5a98: Status 404 returned error can't find the container with id c192190506ed34f4f5fbd11852dd57f0198f0d4f0f56cc95e7be4fc3493d5a98
Jul 07 16:51:53.464478 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:53.464440    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-hvzwl" event={"ID":"7de013cf-4ea5-40a0-b137-4e40ee0db731","Type":"ContainerStarted","Data":"c192190506ed34f4f5fbd11852dd57f0198f0d4f0f56cc95e7be4fc3493d5a98"}
Jul 07 16:51:54.471022 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:54.470970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj" event={"ID":"f3ffcd66-94b9-4f10-b6e2-82f551fe6b65","Type":"ContainerStarted","Data":"58d212901aa1952787716285c2a56ecc7cb5f5e3a5f1415a0a9bb5986c001109"}
Jul 07 16:51:54.471431 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:54.471101    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:51:54.488628 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:54.488577    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj" podStartSLOduration=1.5963935729999998 podStartE2EDuration="3.48855686s" totalImagesPullingTime="1.892163287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:51:54.48742716 +0000 UTC m=+252.750963791" watchObservedRunningTime="2026-07-07 16:51:54.48855686 +0000 UTC m=+252.752093490"
Jul 07 16:51:58.490040 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:58.490003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm" event={"ID":"b5bc9a78-fefc-42a5-9006-43665c1d1494","Type":"ContainerStarted","Data":"b1c6e781f3e8361f3bb0827aa38d23cbd89d166463eb8d7c3632ec05127eb1a1"}
Jul 07 16:51:58.490496 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:58.490098    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:51:58.491409 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:58.491390    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-hvzwl" event={"ID":"7de013cf-4ea5-40a0-b137-4e40ee0db731","Type":"ContainerStarted","Data":"e1ab07401b70b0ec821a4ad43e9b805494c5e8141a9935f334b26929aedda429"}
Jul 07 16:51:58.491479 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:58.491446    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:51:58.506110 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:58.506070    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm" podStartSLOduration=2.584825431 podStartE2EDuration="8.50605908s" totalImagesPullingTime="5.921233649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:51:58.504881151 +0000 UTC m=+256.768417783" watchObservedRunningTime="2026-07-07 16:51:58.50605908 +0000 UTC m=+256.769595709"
Jul 07 16:51:58.521611 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:51:58.521567    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-hvzwl" podStartSLOduration=3.269980433 podStartE2EDuration="8.521551081s" totalImagesPullingTime="5.251570648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:51:58.520821673 +0000 UTC m=+256.784358322" watchObservedRunningTime="2026-07-07 16:51:58.521551081 +0000 UTC m=+256.785087716"
Jul 07 16:52:09.501247 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:09.501174    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-tffqm"
Jul 07 16:52:09.923770 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:09.923732    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:09.983637 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:09.983611    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:11.454335 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:11.454301    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-gsdjl"
Jul 07 16:52:15.478181 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:15.478151    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-vlvfj"
Jul 07 16:52:19.497648 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:19.497620    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-hvzwl"
Jul 07 16:52:42.124855 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:42.124828    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 16:52:44.379285 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:44.379240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:45.877301 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:45.877266    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:56.900783 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.900746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:56.963371 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.963336    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85768989b-mldpj"]
Jul 07 16:52:56.965683 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.965663    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:52:56.968052 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.968032    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 16:52:56.968200 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.968181    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 16:52:56.968960 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.968939    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-5v5p2\""
Jul 07 16:52:56.969059 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.968948    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 07 16:52:56.976389 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.976367    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85768989b-mldpj"]
Jul 07 16:52:56.978701 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.978683    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-92wk6"]
Jul 07 16:52:56.980693 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.980674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:52:56.982733 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.982712    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 07 16:52:56.982840 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.982732    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-blk9s\""
Jul 07 16:52:56.983436 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.983416    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:56.994048 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:56.994019    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-92wk6"]
Jul 07 16:52:57.000625 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.000607    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-5v7z7"]
Jul 07 16:52:57.002728 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.002710    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:52:57.005209 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.005190    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9s599\""
Jul 07 16:52:57.005346 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.005292    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 16:52:57.011321 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.011301    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5v7z7"]
Jul 07 16:52:57.058780 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.058756    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:57.098320 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.098297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-cert\") pod \"kserve-controller-manager-85768989b-mldpj\" (UID: \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\") " pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:52:57.098414 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.098328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pmk\" (UniqueName: \"kubernetes.io/projected/1919256d-0c4b-4f30-9a3c-63c9bde7d35b-kube-api-access-j9pmk\") pod \"llmisvc-controller-manager-67d95fdc48-92wk6\" (UID: \"1919256d-0c4b-4f30-9a3c-63c9bde7d35b\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:52:57.098414 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.098356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q76wm\" (UniqueName: \"kubernetes.io/projected/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-kube-api-access-q76wm\") pod \"kserve-controller-manager-85768989b-mldpj\" (UID: \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\") " pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:52:57.098502 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.098461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1919256d-0c4b-4f30-9a3c-63c9bde7d35b-cert\") pod \"llmisvc-controller-manager-67d95fdc48-92wk6\" (UID: \"1919256d-0c4b-4f30-9a3c-63c9bde7d35b\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:52:57.098502 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.098495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dc7fb89e-0a70-4cf4-a91a-76520ee41aa5-data\") pod \"seaweedfs-64568bcd49-5v7z7\" (UID: \"dc7fb89e-0a70-4cf4-a91a-76520ee41aa5\") " pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:52:57.098579 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.098513    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rsp\" (UniqueName: \"kubernetes.io/projected/dc7fb89e-0a70-4cf4-a91a-76520ee41aa5-kube-api-access-74rsp\") pod \"seaweedfs-64568bcd49-5v7z7\" (UID: \"dc7fb89e-0a70-4cf4-a91a-76520ee41aa5\") " pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:52:57.159434 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.159373    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:57.199503 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.199480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pmk\" (UniqueName: \"kubernetes.io/projected/1919256d-0c4b-4f30-9a3c-63c9bde7d35b-kube-api-access-j9pmk\") pod \"llmisvc-controller-manager-67d95fdc48-92wk6\" (UID: \"1919256d-0c4b-4f30-9a3c-63c9bde7d35b\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:52:57.199584 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.199516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76wm\" (UniqueName: \"kubernetes.io/projected/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-kube-api-access-q76wm\") pod \"kserve-controller-manager-85768989b-mldpj\" (UID: \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\") " pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:52:57.199584 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.199561    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1919256d-0c4b-4f30-9a3c-63c9bde7d35b-cert\") pod \"llmisvc-controller-manager-67d95fdc48-92wk6\" (UID: \"1919256d-0c4b-4f30-9a3c-63c9bde7d35b\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:52:57.199659 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.199592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dc7fb89e-0a70-4cf4-a91a-76520ee41aa5-data\") pod \"seaweedfs-64568bcd49-5v7z7\" (UID: \"dc7fb89e-0a70-4cf4-a91a-76520ee41aa5\") " pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:52:57.199659 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.199614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rsp\" (UniqueName: \"kubernetes.io/projected/dc7fb89e-0a70-4cf4-a91a-76520ee41aa5-kube-api-access-74rsp\") pod \"seaweedfs-64568bcd49-5v7z7\" (UID: \"dc7fb89e-0a70-4cf4-a91a-76520ee41aa5\") " pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:52:57.199659 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.199641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-cert\") pod \"kserve-controller-manager-85768989b-mldpj\" (UID: \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\") " pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:52:57.199780 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:52:57.199689    2559 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 07 16:52:57.199780 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:52:57.199746    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1919256d-0c4b-4f30-9a3c-63c9bde7d35b-cert podName:1919256d-0c4b-4f30-9a3c-63c9bde7d35b nodeName:}" failed. No retries permitted until 2026-07-07 16:52:57.699727235 +0000 UTC m=+315.963263861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1919256d-0c4b-4f30-9a3c-63c9bde7d35b-cert") pod "llmisvc-controller-manager-67d95fdc48-92wk6" (UID: "1919256d-0c4b-4f30-9a3c-63c9bde7d35b") : secret "llmisvc-webhook-server-cert" not found
Jul 07 16:52:57.200021 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.200007    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dc7fb89e-0a70-4cf4-a91a-76520ee41aa5-data\") pod \"seaweedfs-64568bcd49-5v7z7\" (UID: \"dc7fb89e-0a70-4cf4-a91a-76520ee41aa5\") " pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:52:57.201874 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.201857    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-cert\") pod \"kserve-controller-manager-85768989b-mldpj\" (UID: \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\") " pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:52:57.209327 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.209302    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q76wm\" (UniqueName: \"kubernetes.io/projected/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-kube-api-access-q76wm\") pod \"kserve-controller-manager-85768989b-mldpj\" (UID: \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\") " pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:52:57.209412 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.209337    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rsp\" (UniqueName: \"kubernetes.io/projected/dc7fb89e-0a70-4cf4-a91a-76520ee41aa5-kube-api-access-74rsp\") pod \"seaweedfs-64568bcd49-5v7z7\" (UID: \"dc7fb89e-0a70-4cf4-a91a-76520ee41aa5\") " pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:52:57.209450 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.209427    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pmk\" (UniqueName: \"kubernetes.io/projected/1919256d-0c4b-4f30-9a3c-63c9bde7d35b-kube-api-access-j9pmk\") pod \"llmisvc-controller-manager-67d95fdc48-92wk6\" (UID: \"1919256d-0c4b-4f30-9a3c-63c9bde7d35b\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:52:57.259423 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.259397    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:57.275660 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.275642    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:52:57.312733 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.312698    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:52:57.408109 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.408041    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85768989b-mldpj"]
Jul 07 16:52:57.411433 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:52:57.411374    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e9ab7d_9794_4d2f_b5ec_0664fe5e11e2.slice/crio-82fed37563dd94f03aff872e5160a8281218d6444271fa880127836cb7945df9 WatchSource:0}: Error finding container 82fed37563dd94f03aff872e5160a8281218d6444271fa880127836cb7945df9: Status 404 returned error can't find the container with id 82fed37563dd94f03aff872e5160a8281218d6444271fa880127836cb7945df9
Jul 07 16:52:57.412830 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.412811    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:52:57.446719 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.446697    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5v7z7"]
Jul 07 16:52:57.449364 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:52:57.449334    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7fb89e_0a70_4cf4_a91a_76520ee41aa5.slice/crio-76e90469ebe47427ec29e5550c1118f96356cc2e82c2ca72e672e8ca8094a980 WatchSource:0}: Error finding container 76e90469ebe47427ec29e5550c1118f96356cc2e82c2ca72e672e8ca8094a980: Status 404 returned error can't find the container with id 76e90469ebe47427ec29e5550c1118f96356cc2e82c2ca72e672e8ca8094a980
Jul 07 16:52:57.704049 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.703960    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1919256d-0c4b-4f30-9a3c-63c9bde7d35b-cert\") pod \"llmisvc-controller-manager-67d95fdc48-92wk6\" (UID: \"1919256d-0c4b-4f30-9a3c-63c9bde7d35b\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:52:57.706157 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.706140    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1919256d-0c4b-4f30-9a3c-63c9bde7d35b-cert\") pod \"llmisvc-controller-manager-67d95fdc48-92wk6\" (UID: \"1919256d-0c4b-4f30-9a3c-63c9bde7d35b\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:52:57.714301 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.714261    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:57.718565 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.718537    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5v7z7" event={"ID":"dc7fb89e-0a70-4cf4-a91a-76520ee41aa5","Type":"ContainerStarted","Data":"76e90469ebe47427ec29e5550c1118f96356cc2e82c2ca72e672e8ca8094a980"}
Jul 07 16:52:57.719695 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.719676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-mldpj" event={"ID":"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2","Type":"ContainerStarted","Data":"82fed37563dd94f03aff872e5160a8281218d6444271fa880127836cb7945df9"}
Jul 07 16:52:57.814070 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.814038    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:57.866587 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.866561    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:57.891907 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:57.891885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:52:58.039203 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:58.039116    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-92wk6"]
Jul 07 16:52:58.041765 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:52:58.041734    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1919256d_0c4b_4f30_9a3c_63c9bde7d35b.slice/crio-1f014342d1bec29819315f4843505acc383c59ef250582fc68a8d68009e44327 WatchSource:0}: Error finding container 1f014342d1bec29819315f4843505acc383c59ef250582fc68a8d68009e44327: Status 404 returned error can't find the container with id 1f014342d1bec29819315f4843505acc383c59ef250582fc68a8d68009e44327
Jul 07 16:52:58.725819 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:52:58.725777    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6" event={"ID":"1919256d-0c4b-4f30-9a3c-63c9bde7d35b","Type":"ContainerStarted","Data":"1f014342d1bec29819315f4843505acc383c59ef250582fc68a8d68009e44327"}
Jul 07 16:53:02.746929 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:02.746895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5v7z7" event={"ID":"dc7fb89e-0a70-4cf4-a91a-76520ee41aa5","Type":"ContainerStarted","Data":"a3b5e15a39678145bf7a0fc1d5a204a70dcd8c90f727626ea3b3f8067256693e"}
Jul 07 16:53:02.747420 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:02.747011    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:53:02.748495 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:02.748470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-mldpj" event={"ID":"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2","Type":"ContainerStarted","Data":"078ce47189405fb30fffe93f21129794179fd6ba2bb2183192ac1b5a24838549"}
Jul 07 16:53:02.748613 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:02.748513    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:53:02.749778 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:02.749759    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6" event={"ID":"1919256d-0c4b-4f30-9a3c-63c9bde7d35b","Type":"ContainerStarted","Data":"00626d9eb018b8ed1cc60f8f3672a4c493bc53974a422ac9cec2cb8acbadb1f1"}
Jul 07 16:53:02.749995 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:02.749891    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:53:02.762848 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:02.762808    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-5v7z7" podStartSLOduration=2.122592684 podStartE2EDuration="6.762796509s" totalImagesPullingTime="4.640203825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:02.761109127 +0000 UTC m=+321.024645769" watchObservedRunningTime="2026-07-07 16:53:02.762796509 +0000 UTC m=+321.026333142"
Jul 07 16:53:02.776178 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:02.776140    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6" podStartSLOduration=2.67410725 podStartE2EDuration="6.776129208s" totalImagesPullingTime="4.102021958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:02.774739801 +0000 UTC m=+321.038276455" watchObservedRunningTime="2026-07-07 16:53:02.776129208 +0000 UTC m=+321.039665838"
Jul 07 16:53:02.790636 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:02.790601    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85768989b-mldpj" podStartSLOduration=3.11719065 podStartE2EDuration="6.790580999s" totalImagesPullingTime="3.673390349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:02.789106178 +0000 UTC m=+321.052642810" watchObservedRunningTime="2026-07-07 16:53:02.790580999 +0000 UTC m=+321.054117631"
Jul 07 16:53:08.756135 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:08.756107    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-5v7z7"
Jul 07 16:53:33.755687 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:33.755614    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-92wk6"
Jul 07 16:53:33.758539 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:33.758522    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:53:35.668098 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.668059    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85768989b-mldpj"]
Jul 07 16:53:35.668541 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.668295    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-85768989b-mldpj" podUID="e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2" containerName="manager" containerID="cri-o://078ce47189405fb30fffe93f21129794179fd6ba2bb2183192ac1b5a24838549" gracePeriod=10
Jul 07 16:53:35.694390 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.694360    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85768989b-2g66c"]
Jul 07 16:53:35.711128 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.711110    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85768989b-2g66c"]
Jul 07 16:53:35.711207 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.711186    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:53:35.800052 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.800028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ea16fc-9a62-44dc-8ceb-7819773890f9-cert\") pod \"kserve-controller-manager-85768989b-2g66c\" (UID: \"e2ea16fc-9a62-44dc-8ceb-7819773890f9\") " pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:53:35.800156 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.800069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ds9\" (UniqueName: \"kubernetes.io/projected/e2ea16fc-9a62-44dc-8ceb-7819773890f9-kube-api-access-g5ds9\") pod \"kserve-controller-manager-85768989b-2g66c\" (UID: \"e2ea16fc-9a62-44dc-8ceb-7819773890f9\") " pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:53:35.888577 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.888549    2559 generic.go:356] "Generic (PLEG): container finished" podID="e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2" containerID="078ce47189405fb30fffe93f21129794179fd6ba2bb2183192ac1b5a24838549" exitCode=0
Jul 07 16:53:35.888712 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.888607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-mldpj" event={"ID":"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2","Type":"ContainerDied","Data":"078ce47189405fb30fffe93f21129794179fd6ba2bb2183192ac1b5a24838549"}
Jul 07 16:53:35.900910 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.900885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ea16fc-9a62-44dc-8ceb-7819773890f9-cert\") pod \"kserve-controller-manager-85768989b-2g66c\" (UID: \"e2ea16fc-9a62-44dc-8ceb-7819773890f9\") " pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:53:35.901046 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.900927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ds9\" (UniqueName: \"kubernetes.io/projected/e2ea16fc-9a62-44dc-8ceb-7819773890f9-kube-api-access-g5ds9\") pod \"kserve-controller-manager-85768989b-2g66c\" (UID: \"e2ea16fc-9a62-44dc-8ceb-7819773890f9\") " pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:53:35.903258 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.903235    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ea16fc-9a62-44dc-8ceb-7819773890f9-cert\") pod \"kserve-controller-manager-85768989b-2g66c\" (UID: \"e2ea16fc-9a62-44dc-8ceb-7819773890f9\") " pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:53:35.909357 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.909334    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ds9\" (UniqueName: \"kubernetes.io/projected/e2ea16fc-9a62-44dc-8ceb-7819773890f9-kube-api-access-g5ds9\") pod \"kserve-controller-manager-85768989b-2g66c\" (UID: \"e2ea16fc-9a62-44dc-8ceb-7819773890f9\") " pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:53:35.945287 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:35.945239    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:53:36.063204 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.063174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:53:36.102592 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.102566    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-cert\") pod \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\" (UID: \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\") "
Jul 07 16:53:36.102734 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.102607    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q76wm\" (UniqueName: \"kubernetes.io/projected/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-kube-api-access-q76wm\") pod \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\" (UID: \"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2\") "
Jul 07 16:53:36.104633 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.104606    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-cert" (OuterVolumeSpecName: "cert") pod "e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2" (UID: "e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:53:36.104760 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.104681    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-kube-api-access-q76wm" (OuterVolumeSpecName: "kube-api-access-q76wm") pod "e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2" (UID: "e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2"). InnerVolumeSpecName "kube-api-access-q76wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:53:36.200625 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.200551    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85768989b-2g66c"]
Jul 07 16:53:36.203549 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.203527    2559 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-cert\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:53:36.203653 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.203558    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q76wm\" (UniqueName: \"kubernetes.io/projected/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2-kube-api-access-q76wm\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:53:36.203749 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:53:36.203726    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ea16fc_9a62_44dc_8ceb_7819773890f9.slice/crio-e9ed76449ae8e76fbcb0faa2c48c33f2ea71531c45f8613474a80a051cbc34ad WatchSource:0}: Error finding container e9ed76449ae8e76fbcb0faa2c48c33f2ea71531c45f8613474a80a051cbc34ad: Status 404 returned error can't find the container with id e9ed76449ae8e76fbcb0faa2c48c33f2ea71531c45f8613474a80a051cbc34ad
Jul 07 16:53:36.893667 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.893642    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-mldpj"
Jul 07 16:53:36.894108 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.893701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-mldpj" event={"ID":"e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2","Type":"ContainerDied","Data":"82fed37563dd94f03aff872e5160a8281218d6444271fa880127836cb7945df9"}
Jul 07 16:53:36.894108 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.893757    2559 scope.go:117] "RemoveContainer" containerID="078ce47189405fb30fffe93f21129794179fd6ba2bb2183192ac1b5a24838549"
Jul 07 16:53:36.895334 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.895309    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-2g66c" event={"ID":"e2ea16fc-9a62-44dc-8ceb-7819773890f9","Type":"ContainerStarted","Data":"b6b2302a5c4bdc29a1277a0fd1c16a05eb09088d0d5d39ac1eec514524cb259f"}
Jul 07 16:53:36.895471 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.895357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-2g66c" event={"ID":"e2ea16fc-9a62-44dc-8ceb-7819773890f9","Type":"ContainerStarted","Data":"e9ed76449ae8e76fbcb0faa2c48c33f2ea71531c45f8613474a80a051cbc34ad"}
Jul 07 16:53:36.895471 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.895388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:53:36.911258 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.911220    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85768989b-2g66c" podStartSLOduration=1.346307904 podStartE2EDuration="1.911207888s" totalImagesPullingTime="564.899984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:36.910124912 +0000 UTC m=+355.173661559" watchObservedRunningTime="2026-07-07 16:53:36.911207888 +0000 UTC m=+355.174744523"
Jul 07 16:53:36.923788 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.923766    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85768989b-mldpj"]
Jul 07 16:53:36.927563 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:36.927540    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-85768989b-mldpj"]
Jul 07 16:53:37.335996 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:53:37.335956    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2" path="/var/lib/kubelet/pods/e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2/volumes"
Jul 07 16:54:02.527187 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:02.527152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:02.581102 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:02.581072    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:07.909652 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:07.909624    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85768989b-2g66c"
Jul 07 16:54:09.026581 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.026545    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:09.098776 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.098750    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zp428"]
Jul 07 16:54:09.099123 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.099108    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2" containerName="manager"
Jul 07 16:54:09.099175 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.099125    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2" containerName="manager"
Jul 07 16:54:09.099209 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.099186    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e9ab7d-9794-4d2f-b5ec-0664fe5e11e2" containerName="manager"
Jul 07 16:54:09.101351 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.101333    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:09.103640 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.103620    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 07 16:54:09.103640 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.103630    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-pltbr\""
Jul 07 16:54:09.105661 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.105637    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:09.109858 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.109830    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zp428"]
Jul 07 16:54:09.163044 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.163017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc00f23-8c4e-46c2-a660-010ec659be5c-cert\") pod \"odh-model-controller-84b54b7f97-zp428\" (UID: \"bcc00f23-8c4e-46c2-a660-010ec659be5c\") " pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:09.163136 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.163075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvfk\" (UniqueName: \"kubernetes.io/projected/bcc00f23-8c4e-46c2-a660-010ec659be5c-kube-api-access-5rvfk\") pod \"odh-model-controller-84b54b7f97-zp428\" (UID: \"bcc00f23-8c4e-46c2-a660-010ec659be5c\") " pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:09.236104 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.236080    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:09.263730 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.263706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvfk\" (UniqueName: \"kubernetes.io/projected/bcc00f23-8c4e-46c2-a660-010ec659be5c-kube-api-access-5rvfk\") pod \"odh-model-controller-84b54b7f97-zp428\" (UID: \"bcc00f23-8c4e-46c2-a660-010ec659be5c\") " pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:09.263812 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.263750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc00f23-8c4e-46c2-a660-010ec659be5c-cert\") pod \"odh-model-controller-84b54b7f97-zp428\" (UID: \"bcc00f23-8c4e-46c2-a660-010ec659be5c\") " pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:09.263850 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:54:09.263836    2559 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 07 16:54:09.263902 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:54:09.263893    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcc00f23-8c4e-46c2-a660-010ec659be5c-cert podName:bcc00f23-8c4e-46c2-a660-010ec659be5c nodeName:}" failed. No retries permitted until 2026-07-07 16:54:09.763879852 +0000 UTC m=+388.027416461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bcc00f23-8c4e-46c2-a660-010ec659be5c-cert") pod "odh-model-controller-84b54b7f97-zp428" (UID: "bcc00f23-8c4e-46c2-a660-010ec659be5c") : secret "odh-model-controller-webhook-cert" not found
Jul 07 16:54:09.272382 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.272359    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvfk\" (UniqueName: \"kubernetes.io/projected/bcc00f23-8c4e-46c2-a660-010ec659be5c-kube-api-access-5rvfk\") pod \"odh-model-controller-84b54b7f97-zp428\" (UID: \"bcc00f23-8c4e-46c2-a660-010ec659be5c\") " pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:09.373375 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.373305    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:09.748390 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.748300    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:09.768454 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.768430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc00f23-8c4e-46c2-a660-010ec659be5c-cert\") pod \"odh-model-controller-84b54b7f97-zp428\" (UID: \"bcc00f23-8c4e-46c2-a660-010ec659be5c\") " pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:09.770797 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.770776    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcc00f23-8c4e-46c2-a660-010ec659be5c-cert\") pod \"odh-model-controller-84b54b7f97-zp428\" (UID: \"bcc00f23-8c4e-46c2-a660-010ec659be5c\") " pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:09.809899 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:09.809876    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:10.013059 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:10.013027    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:10.137016 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:10.136814    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-zp428"]
Jul 07 16:54:10.139762 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:54:10.139735    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc00f23_8c4e_46c2_a660_010ec659be5c.slice/crio-d17d165ef3926f5dbf71fcb0b41c6c0355fcac6caef88d0b428b1dce29ed0f10 WatchSource:0}: Error finding container d17d165ef3926f5dbf71fcb0b41c6c0355fcac6caef88d0b428b1dce29ed0f10: Status 404 returned error can't find the container with id d17d165ef3926f5dbf71fcb0b41c6c0355fcac6caef88d0b428b1dce29ed0f10
Jul 07 16:54:11.049369 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:11.049332    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zp428" event={"ID":"bcc00f23-8c4e-46c2-a660-010ec659be5c","Type":"ContainerStarted","Data":"d17d165ef3926f5dbf71fcb0b41c6c0355fcac6caef88d0b428b1dce29ed0f10"}
Jul 07 16:54:13.060364 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:13.060331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-zp428" event={"ID":"bcc00f23-8c4e-46c2-a660-010ec659be5c","Type":"ContainerStarted","Data":"6e030363092ba6d4f4559b995737bf0cd3946fc9d4eadc03f1d9a96d405a63dc"}
Jul 07 16:54:13.060732 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:13.060485    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:13.075556 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:13.075516    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-zp428" podStartSLOduration=1.346659599 podStartE2EDuration="4.07550252s" totalImagesPullingTime="2.728842921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:54:13.074394812 +0000 UTC m=+391.337931447" watchObservedRunningTime="2026-07-07 16:54:13.07550252 +0000 UTC m=+391.339039151"
Jul 07 16:54:13.336115 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:13.336083    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dd7479777-g85ns"]
Jul 07 16:54:13.443042 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:13.443000    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:13.673738 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:13.673665    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:13.774635 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:13.774605    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:24.066744 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:24.066717    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-zp428"
Jul 07 16:54:38.356999 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.356927    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7dd7479777-g85ns" podUID="5546ee6a-e789-4744-a278-c78c4a7b71bb" containerName="console" containerID="cri-o://4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968" gracePeriod=15
Jul 07 16:54:38.599451 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.599428    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:54:38.696417 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.696345    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-config\") pod \"5546ee6a-e789-4744-a278-c78c4a7b71bb\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") "
Jul 07 16:54:38.696417 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.696377    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-service-ca\") pod \"5546ee6a-e789-4744-a278-c78c4a7b71bb\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") "
Jul 07 16:54:38.696417 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.696401    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crv6p\" (UniqueName: \"kubernetes.io/projected/5546ee6a-e789-4744-a278-c78c4a7b71bb-kube-api-access-crv6p\") pod \"5546ee6a-e789-4744-a278-c78c4a7b71bb\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") "
Jul 07 16:54:38.696699 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.696424    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-trusted-ca-bundle\") pod \"5546ee6a-e789-4744-a278-c78c4a7b71bb\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") "
Jul 07 16:54:38.696699 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.696445    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-oauth-serving-cert\") pod \"5546ee6a-e789-4744-a278-c78c4a7b71bb\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") "
Jul 07 16:54:38.696699 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.696500    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-serving-cert\") pod \"5546ee6a-e789-4744-a278-c78c4a7b71bb\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") "
Jul 07 16:54:38.696699 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.696516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-oauth-config\") pod \"5546ee6a-e789-4744-a278-c78c4a7b71bb\" (UID: \"5546ee6a-e789-4744-a278-c78c4a7b71bb\") "
Jul 07 16:54:38.696898 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.696753    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-config" (OuterVolumeSpecName: "console-config") pod "5546ee6a-e789-4744-a278-c78c4a7b71bb" (UID: "5546ee6a-e789-4744-a278-c78c4a7b71bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:54:38.696955 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.696919    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5546ee6a-e789-4744-a278-c78c4a7b71bb" (UID: "5546ee6a-e789-4744-a278-c78c4a7b71bb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:54:38.697094 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.697069    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "5546ee6a-e789-4744-a278-c78c4a7b71bb" (UID: "5546ee6a-e789-4744-a278-c78c4a7b71bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:54:38.697156 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.697125    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5546ee6a-e789-4744-a278-c78c4a7b71bb" (UID: "5546ee6a-e789-4744-a278-c78c4a7b71bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:54:38.698685 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.698654    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5546ee6a-e789-4744-a278-c78c4a7b71bb-kube-api-access-crv6p" (OuterVolumeSpecName: "kube-api-access-crv6p") pod "5546ee6a-e789-4744-a278-c78c4a7b71bb" (UID: "5546ee6a-e789-4744-a278-c78c4a7b71bb"). InnerVolumeSpecName "kube-api-access-crv6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:54:38.699085 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.699064    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5546ee6a-e789-4744-a278-c78c4a7b71bb" (UID: "5546ee6a-e789-4744-a278-c78c4a7b71bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:54:38.699145 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.699118    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5546ee6a-e789-4744-a278-c78c4a7b71bb" (UID: "5546ee6a-e789-4744-a278-c78c4a7b71bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:54:38.797190 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.797155    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-oauth-serving-cert\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:54:38.797190 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.797184    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-serving-cert\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:54:38.797190 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.797193    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-oauth-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:54:38.797411 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.797205    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-console-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:54:38.797411 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.797214    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-service-ca\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:54:38.797411 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.797221    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crv6p\" (UniqueName: \"kubernetes.io/projected/5546ee6a-e789-4744-a278-c78c4a7b71bb-kube-api-access-crv6p\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:54:38.797411 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:38.797230    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5546ee6a-e789-4744-a278-c78c4a7b71bb-trusted-ca-bundle\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:54:39.174445 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.174410    2559 generic.go:356] "Generic (PLEG): container finished" podID="5546ee6a-e789-4744-a278-c78c4a7b71bb" containerID="4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968" exitCode=0
Jul 07 16:54:39.174624 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.174459    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd7479777-g85ns" event={"ID":"5546ee6a-e789-4744-a278-c78c4a7b71bb","Type":"ContainerDied","Data":"4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968"}
Jul 07 16:54:39.174624 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.174498    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd7479777-g85ns" event={"ID":"5546ee6a-e789-4744-a278-c78c4a7b71bb","Type":"ContainerDied","Data":"fe5aeb5addd233e8154235cc1d0ee94d1d1512cf9b15be4c7afba6950be6b36a"}
Jul 07 16:54:39.174624 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.174522    2559 scope.go:117] "RemoveContainer" containerID="4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968"
Jul 07 16:54:39.174624 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.174459    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd7479777-g85ns"
Jul 07 16:54:39.184794 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.184692    2559 scope.go:117] "RemoveContainer" containerID="4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968"
Jul 07 16:54:39.185180 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:54:39.185151    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968\": container with ID starting with 4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968 not found: ID does not exist" containerID="4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968"
Jul 07 16:54:39.185252 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.185216    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968"} err="failed to get container status \"4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968\": rpc error: code = NotFound desc = could not find container \"4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968\": container with ID starting with 4f304653f06bf66da97eeccf1b76b257b6607b867d66b17c832f5808fe7a8968 not found: ID does not exist"
Jul 07 16:54:39.199952 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.199913    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dd7479777-g85ns"]
Jul 07 16:54:39.201506 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.201476    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dd7479777-g85ns"]
Jul 07 16:54:39.337289 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:39.337254    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5546ee6a-e789-4744-a278-c78c4a7b71bb" path="/var/lib/kubelet/pods/5546ee6a-e789-4744-a278-c78c4a7b71bb/volumes"
Jul 07 16:54:49.506074 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:49.506032    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:49.557392 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:54:49.557351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:29.089525 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:29.089483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:43.004405 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:43.004369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:43.106299 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:43.106263    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:54.698516 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.698485    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:54.887496 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.887460    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:54.931928 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.931888    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"]
Jul 07 16:55:54.932313 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.932295    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5546ee6a-e789-4744-a278-c78c4a7b71bb" containerName="console"
Jul 07 16:55:54.932313 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.932315    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5546ee6a-e789-4744-a278-c78c4a7b71bb" containerName="console"
Jul 07 16:55:54.932516 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.932448    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5546ee6a-e789-4744-a278-c78c4a7b71bb" containerName="console"
Jul 07 16:55:54.934644 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.934624    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:54.938290 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.938265    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ae852-kube-rbac-proxy-sar-config\""
Jul 07 16:55:54.938562 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.938540    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ae852-predictor-serving-cert\""
Jul 07 16:55:54.938654 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.938378    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 16:55:54.938654 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.938619    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lvhm6\""
Jul 07 16:55:54.939131 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.939110    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 16:55:54.950185 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.950116    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"]
Jul 07 16:55:54.994613 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:54.994577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:55.071454 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.071388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:55.086314 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.086289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9x8\" (UniqueName: \"kubernetes.io/projected/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-api-access-cf9x8\") pod \"success-200-isvc-ae852-predictor-7dcbf55c45-drv4l\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") " pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.086448 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.086325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ae852-predictor-7dcbf55c45-drv4l\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") " pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.086448 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.086366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-proxy-tls\") pod \"success-200-isvc-ae852-predictor-7dcbf55c45-drv4l\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") " pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.131365 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.131332    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:55.187605 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.187569    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9x8\" (UniqueName: \"kubernetes.io/projected/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-api-access-cf9x8\") pod \"success-200-isvc-ae852-predictor-7dcbf55c45-drv4l\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") " pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.187760 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.187609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ae852-predictor-7dcbf55c45-drv4l\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") " pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.187760 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.187647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-proxy-tls\") pod \"success-200-isvc-ae852-predictor-7dcbf55c45-drv4l\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") " pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.188263 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.188244    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ae852-predictor-7dcbf55c45-drv4l\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") " pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.190128 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.190107    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-proxy-tls\") pod \"success-200-isvc-ae852-predictor-7dcbf55c45-drv4l\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") " pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.196841 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.196822    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9x8\" (UniqueName: \"kubernetes.io/projected/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-api-access-cf9x8\") pod \"success-200-isvc-ae852-predictor-7dcbf55c45-drv4l\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") " pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.206480 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.206409    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:55.251348 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.251320    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:55:55.274257 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.274233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:55.386454 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.386427    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"]
Jul 07 16:55:55.390318 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:55:55.390282    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7eb2e6e_7623_4809_b4a7_bb46ec9c6360.slice/crio-71490f09c12b436ca838b509f6116178f9d324e3a96b224b85bd5d7f86e5f1c1 WatchSource:0}: Error finding container 71490f09c12b436ca838b509f6116178f9d324e3a96b224b85bd5d7f86e5f1c1: Status 404 returned error can't find the container with id 71490f09c12b436ca838b509f6116178f9d324e3a96b224b85bd5d7f86e5f1c1
Jul 07 16:55:55.390502 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.390478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:55.487175 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.487096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" event={"ID":"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360","Type":"ContainerStarted","Data":"71490f09c12b436ca838b509f6116178f9d324e3a96b224b85bd5d7f86e5f1c1"}
Jul 07 16:55:55.620758 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:55.620720    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:58.432787 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:55:58.432742    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:56:09.567737 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:09.567691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" event={"ID":"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360","Type":"ContainerStarted","Data":"a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820"}
Jul 07 16:56:11.581289 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:11.581253    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" event={"ID":"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360","Type":"ContainerStarted","Data":"82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e"}
Jul 07 16:56:11.581702 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:11.581461    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:56:11.581702 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:11.581593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:56:11.583039 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:11.583009    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 07 16:56:11.597725 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:11.597684    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" podStartSLOduration=2.010926119 podStartE2EDuration="17.597671572s" totalImagesPullingTime="15.586745453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:11.596609653 +0000 UTC m=+509.860146304" watchObservedRunningTime="2026-07-07 16:56:11.597671572 +0000 UTC m=+509.861208203"
Jul 07 16:56:12.585359 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:12.585318    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 07 16:56:17.590116 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:17.590089    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:56:17.590522 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:17.590497    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 07 16:56:27.591561 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:27.591516    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 07 16:56:37.590927 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:37.590889    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 07 16:56:47.590777 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:47.590730    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 07 16:56:57.591666 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:56:57.591639    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:57:10.117265 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:10.117226    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:24.735489 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.732082    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"]
Jul 07 16:57:24.736929 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.736906    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:24.737769 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.737720    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:24.739468 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.739442    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ae852-kube-rbac-proxy-sar-config\""
Jul 07 16:57:24.739688 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.739667    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ae852-serving-cert\""
Jul 07 16:57:24.740537 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.740515    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"]
Jul 07 16:57:24.786504 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.786480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03a0acd-e690-4085-ba36-3e3b148aae0a-proxy-tls\") pod \"switch-graph-ae852-588c44875-cd7jp\" (UID: \"b03a0acd-e690-4085-ba36-3e3b148aae0a\") " pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:24.786590 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.786515    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03a0acd-e690-4085-ba36-3e3b148aae0a-openshift-service-ca-bundle\") pod \"switch-graph-ae852-588c44875-cd7jp\" (UID: \"b03a0acd-e690-4085-ba36-3e3b148aae0a\") " pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:24.887383 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.887361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03a0acd-e690-4085-ba36-3e3b148aae0a-proxy-tls\") pod \"switch-graph-ae852-588c44875-cd7jp\" (UID: \"b03a0acd-e690-4085-ba36-3e3b148aae0a\") " pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:24.887476 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.887399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03a0acd-e690-4085-ba36-3e3b148aae0a-openshift-service-ca-bundle\") pod \"switch-graph-ae852-588c44875-cd7jp\" (UID: \"b03a0acd-e690-4085-ba36-3e3b148aae0a\") " pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:24.887522 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:57:24.887498    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-ae852-serving-cert: secret "switch-graph-ae852-serving-cert" not found
Jul 07 16:57:24.887570 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:57:24.887558    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b03a0acd-e690-4085-ba36-3e3b148aae0a-proxy-tls podName:b03a0acd-e690-4085-ba36-3e3b148aae0a nodeName:}" failed. No retries permitted until 2026-07-07 16:57:25.387539907 +0000 UTC m=+583.651076520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b03a0acd-e690-4085-ba36-3e3b148aae0a-proxy-tls") pod "switch-graph-ae852-588c44875-cd7jp" (UID: "b03a0acd-e690-4085-ba36-3e3b148aae0a") : secret "switch-graph-ae852-serving-cert" not found
Jul 07 16:57:24.887966 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.887947    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03a0acd-e690-4085-ba36-3e3b148aae0a-openshift-service-ca-bundle\") pod \"switch-graph-ae852-588c44875-cd7jp\" (UID: \"b03a0acd-e690-4085-ba36-3e3b148aae0a\") " pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:24.939511 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:24.939487    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:25.392676 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:25.392646    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03a0acd-e690-4085-ba36-3e3b148aae0a-proxy-tls\") pod \"switch-graph-ae852-588c44875-cd7jp\" (UID: \"b03a0acd-e690-4085-ba36-3e3b148aae0a\") " pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:25.395088 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:25.395065    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03a0acd-e690-4085-ba36-3e3b148aae0a-proxy-tls\") pod \"switch-graph-ae852-588c44875-cd7jp\" (UID: \"b03a0acd-e690-4085-ba36-3e3b148aae0a\") " pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:25.650297 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:25.650212    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:25.773627 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:25.773603    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"]
Jul 07 16:57:25.775883 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:57:25.775854    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03a0acd_e690_4085_ba36_3e3b148aae0a.slice/crio-f8051916057aaa646c23752d20c5365ec0f09058173a02cf0b8b51124e80e5a4 WatchSource:0}: Error finding container f8051916057aaa646c23752d20c5365ec0f09058173a02cf0b8b51124e80e5a4: Status 404 returned error can't find the container with id f8051916057aaa646c23752d20c5365ec0f09058173a02cf0b8b51124e80e5a4
Jul 07 16:57:25.893598 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:25.893566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" event={"ID":"b03a0acd-e690-4085-ba36-3e3b148aae0a","Type":"ContainerStarted","Data":"f8051916057aaa646c23752d20c5365ec0f09058173a02cf0b8b51124e80e5a4"}
Jul 07 16:57:29.913691 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:29.913658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" event={"ID":"b03a0acd-e690-4085-ba36-3e3b148aae0a","Type":"ContainerStarted","Data":"e6fc3572e8c326ac8aa09aee664672e5a514febda6ed32b300c70abbf982fe2f"}
Jul 07 16:57:29.914110 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:29.913732    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:29.929637 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:29.929590    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" podStartSLOduration=2.7499755219999997 podStartE2EDuration="5.929575588s" totalImagesPullingTime="3.179600066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:57:29.927621979 +0000 UTC m=+588.191158612" watchObservedRunningTime="2026-07-07 16:57:29.929575588 +0000 UTC m=+588.193112213"
Jul 07 16:57:35.928679 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:35.928652    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:38.990366 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:38.990333    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"]
Jul 07 16:57:38.990778 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:38.990572    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852" containerID="cri-o://e6fc3572e8c326ac8aa09aee664672e5a514febda6ed32b300c70abbf982fe2f" gracePeriod=30
Jul 07 16:57:39.013056 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.013017    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:39.186237 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.186200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:39.228196 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.228167    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"]
Jul 07 16:57:39.230914 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.230895    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.233490 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.233471    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bb9ec-kube-rbac-proxy-sar-config\""
Jul 07 16:57:39.233603 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.233498    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bb9ec-predictor-serving-cert\""
Jul 07 16:57:39.244747 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.244689    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"]
Jul 07 16:57:39.246383 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.246354    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"]
Jul 07 16:57:39.246636 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.246608    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container" containerID="cri-o://a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820" gracePeriod=30
Jul 07 16:57:39.246738 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.246707    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kube-rbac-proxy" containerID="cri-o://82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e" gracePeriod=30
Jul 07 16:57:39.276896 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.276863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:39.296320 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.296293    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73352ca-099a-4c85-b42f-af052a96caa1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") " pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.296424 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.296339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73352ca-099a-4c85-b42f-af052a96caa1-proxy-tls\") pod \"success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") " pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.296473 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.296432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84km5\" (UniqueName: \"kubernetes.io/projected/c73352ca-099a-4c85-b42f-af052a96caa1-kube-api-access-84km5\") pod \"success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") " pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.347194 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.347162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:39.397276 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.397248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73352ca-099a-4c85-b42f-af052a96caa1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") " pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.397395 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.397290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73352ca-099a-4c85-b42f-af052a96caa1-proxy-tls\") pod \"success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") " pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.397395 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.397355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84km5\" (UniqueName: \"kubernetes.io/projected/c73352ca-099a-4c85-b42f-af052a96caa1-kube-api-access-84km5\") pod \"success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") " pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.397914 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.397894    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73352ca-099a-4c85-b42f-af052a96caa1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") " pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.399467 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.399446    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73352ca-099a-4c85-b42f-af052a96caa1-proxy-tls\") pod \"success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") " pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.403248 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.403201    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:39.405381 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.405355    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84km5\" (UniqueName: \"kubernetes.io/projected/c73352ca-099a-4c85-b42f-af052a96caa1-kube-api-access-84km5\") pod \"success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") " pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.503945 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.503920    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:39.541980 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.541955    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.664925 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.664897    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"]
Jul 07 16:57:39.667751 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:57:39.667720    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73352ca_099a_4c85_b42f_af052a96caa1.slice/crio-ca32d8167ae470fa2dd29f9042727cb7808ef5aed2042e167c193ff25dd8f7c0 WatchSource:0}: Error finding container ca32d8167ae470fa2dd29f9042727cb7808ef5aed2042e167c193ff25dd8f7c0: Status 404 returned error can't find the container with id ca32d8167ae470fa2dd29f9042727cb7808ef5aed2042e167c193ff25dd8f7c0
Jul 07 16:57:39.964407 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.964375    2559 generic.go:356] "Generic (PLEG): container finished" podID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerID="82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e" exitCode=2
Jul 07 16:57:39.964586 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.964455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" event={"ID":"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360","Type":"ContainerDied","Data":"82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e"}
Jul 07 16:57:39.966058 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.966037    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" event={"ID":"c73352ca-099a-4c85-b42f-af052a96caa1","Type":"ContainerStarted","Data":"45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672"}
Jul 07 16:57:39.966191 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.966079    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" event={"ID":"c73352ca-099a-4c85-b42f-af052a96caa1","Type":"ContainerStarted","Data":"d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074"}
Jul 07 16:57:39.966191 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.966098    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" event={"ID":"c73352ca-099a-4c85-b42f-af052a96caa1","Type":"ContainerStarted","Data":"ca32d8167ae470fa2dd29f9042727cb7808ef5aed2042e167c193ff25dd8f7c0"}
Jul 07 16:57:39.966306 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.966202    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:39.983188 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:39.983140    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podStartSLOduration=0.983126679 podStartE2EDuration="983.126679ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:57:39.981559672 +0000 UTC m=+598.245096309" watchObservedRunningTime="2026-07-07 16:57:39.983126679 +0000 UTC m=+598.246663310"
Jul 07 16:57:40.926468 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:40.926434    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:40.970563 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:40.970535    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:40.971941 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:40.971916    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 16:57:41.974542 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:41.974502    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 16:57:42.302287 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.302257    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:57:42.422219 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.422192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-proxy-tls\") pod \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") "
Jul 07 16:57:42.422379 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.422240    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf9x8\" (UniqueName: \"kubernetes.io/projected/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-api-access-cf9x8\") pod \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") "
Jul 07 16:57:42.422379 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.422274    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-rbac-proxy-sar-config\") pod \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\" (UID: \"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360\") "
Jul 07 16:57:42.422760 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.422727    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" (UID: "b7eb2e6e-7623-4809-b4a7-bb46ec9c6360"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:57:42.424331 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.424306    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-api-access-cf9x8" (OuterVolumeSpecName: "kube-api-access-cf9x8") pod "b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" (UID: "b7eb2e6e-7623-4809-b4a7-bb46ec9c6360"). InnerVolumeSpecName "kube-api-access-cf9x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:57:42.424423 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.424326    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" (UID: "b7eb2e6e-7623-4809-b4a7-bb46ec9c6360"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:57:42.523527 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.523499    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:57:42.523527 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.523524    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:57:42.523670 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.523535    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9x8\" (UniqueName: \"kubernetes.io/projected/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360-kube-api-access-cf9x8\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:57:42.980949 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.980918    2559 generic.go:356] "Generic (PLEG): container finished" podID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerID="a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820" exitCode=0
Jul 07 16:57:42.981420 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.980965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" event={"ID":"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360","Type":"ContainerDied","Data":"a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820"}
Jul 07 16:57:42.981420 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.980970    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"
Jul 07 16:57:42.981420 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.981032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l" event={"ID":"b7eb2e6e-7623-4809-b4a7-bb46ec9c6360","Type":"ContainerDied","Data":"71490f09c12b436ca838b509f6116178f9d324e3a96b224b85bd5d7f86e5f1c1"}
Jul 07 16:57:42.981420 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.981058    2559 scope.go:117] "RemoveContainer" containerID="82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e"
Jul 07 16:57:42.989429 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:42.989414    2559 scope.go:117] "RemoveContainer" containerID="a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820"
Jul 07 16:57:43.001473 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:43.001449    2559 scope.go:117] "RemoveContainer" containerID="82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e"
Jul 07 16:57:43.001765 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:57:43.001742    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e\": container with ID starting with 82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e not found: ID does not exist" containerID="82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e"
Jul 07 16:57:43.001829 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:43.001771    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e"} err="failed to get container status \"82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e\": rpc error: code = NotFound desc = could not find container \"82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e\": container with ID starting with 82d38dc2d5ce6d2ae38aa5113e9f34bd2a89e3cd5aec6c0772a17a060f73cf6e not found: ID does not exist"
Jul 07 16:57:43.001829 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:43.001788    2559 scope.go:117] "RemoveContainer" containerID="a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820"
Jul 07 16:57:43.002051 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:57:43.002034    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820\": container with ID starting with a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820 not found: ID does not exist" containerID="a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820"
Jul 07 16:57:43.002101 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:43.002055    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820"} err="failed to get container status \"a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820\": rpc error: code = NotFound desc = could not find container \"a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820\": container with ID starting with a3a84bbe200ba1c67000bb23878d974f4979cdf969e537b896f9ed4e525d6820 not found: ID does not exist"
Jul 07 16:57:43.002152 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:43.002136    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"]
Jul 07 16:57:43.007290 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:43.007271    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ae852-predictor-7dcbf55c45-drv4l"]
Jul 07 16:57:43.336833 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:43.336794    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" path="/var/lib/kubelet/pods/b7eb2e6e-7623-4809-b4a7-bb46ec9c6360/volumes"
Jul 07 16:57:45.927335 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:45.927298    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:46.978863 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:46.978838    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:57:46.979366 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:46.979340    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 16:57:50.927395 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:50.927352    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:50.927754 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:50.927453    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:57:55.927237 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:55.927191    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:56.980204 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:57:56.980161    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 16:58:00.927253 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:00.927215    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:04.699561 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.699487    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"]
Jul 07 16:58:04.699892 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.699825    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container"
Jul 07 16:58:04.699892 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.699863    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container"
Jul 07 16:58:04.699892 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.699879    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kube-rbac-proxy"
Jul 07 16:58:04.699892 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.699885    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kube-rbac-proxy"
Jul 07 16:58:04.700048 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.699944    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kserve-container"
Jul 07 16:58:04.700048 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.699954    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7eb2e6e-7623-4809-b4a7-bb46ec9c6360" containerName="kube-rbac-proxy"
Jul 07 16:58:04.703209 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.703192    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:04.705279 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.705259    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 07 16:58:04.705449 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.705428    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 07 16:58:04.706790 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.706753    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:04.711027 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.711006    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"]
Jul 07 16:58:04.804817 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.804783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115662c5-a931-43d7-b818-d9d71524937d-proxy-tls\") pod \"model-chainer-565d7b4557-w6klz\" (UID: \"115662c5-a931-43d7-b818-d9d71524937d\") " pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:04.804945 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.804891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115662c5-a931-43d7-b818-d9d71524937d-openshift-service-ca-bundle\") pod \"model-chainer-565d7b4557-w6klz\" (UID: \"115662c5-a931-43d7-b818-d9d71524937d\") " pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:04.905673 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.905647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115662c5-a931-43d7-b818-d9d71524937d-openshift-service-ca-bundle\") pod \"model-chainer-565d7b4557-w6klz\" (UID: \"115662c5-a931-43d7-b818-d9d71524937d\") " pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:04.905803 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.905690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115662c5-a931-43d7-b818-d9d71524937d-proxy-tls\") pod \"model-chainer-565d7b4557-w6klz\" (UID: \"115662c5-a931-43d7-b818-d9d71524937d\") " pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:04.906307 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.906284    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115662c5-a931-43d7-b818-d9d71524937d-openshift-service-ca-bundle\") pod \"model-chainer-565d7b4557-w6klz\" (UID: \"115662c5-a931-43d7-b818-d9d71524937d\") " pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:04.908116 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:04.908096    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115662c5-a931-43d7-b818-d9d71524937d-proxy-tls\") pod \"model-chainer-565d7b4557-w6klz\" (UID: \"115662c5-a931-43d7-b818-d9d71524937d\") " pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:05.014807 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:05.014781    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:05.138420 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:05.138398    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"]
Jul 07 16:58:05.140427 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:58:05.140400    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115662c5_a931_43d7_b818_d9d71524937d.slice/crio-a9052ae71cfded379c8a8ae2b4fb4ac8a3b9240fa31708666ba1672bb214ffec WatchSource:0}: Error finding container a9052ae71cfded379c8a8ae2b4fb4ac8a3b9240fa31708666ba1672bb214ffec: Status 404 returned error can't find the container with id a9052ae71cfded379c8a8ae2b4fb4ac8a3b9240fa31708666ba1672bb214ffec
Jul 07 16:58:05.927249 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:05.927213    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:05.965434 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:05.965402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:06.084114 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:06.084081    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" event={"ID":"115662c5-a931-43d7-b818-d9d71524937d","Type":"ContainerStarted","Data":"62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239"}
Jul 07 16:58:06.084284 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:06.084136    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:06.084284 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:06.084158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" event={"ID":"115662c5-a931-43d7-b818-d9d71524937d","Type":"ContainerStarted","Data":"a9052ae71cfded379c8a8ae2b4fb4ac8a3b9240fa31708666ba1672bb214ffec"}
Jul 07 16:58:06.100257 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:06.100211    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" podStartSLOduration=2.100197666 podStartE2EDuration="2.100197666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:58:06.09864472 +0000 UTC m=+624.362181368" watchObservedRunningTime="2026-07-07 16:58:06.100197666 +0000 UTC m=+624.363734341"
Jul 07 16:58:06.979319 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:06.979280    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 16:58:09.099326 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:09.099294    2559 generic.go:356] "Generic (PLEG): container finished" podID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerID="e6fc3572e8c326ac8aa09aee664672e5a514febda6ed32b300c70abbf982fe2f" exitCode=0
Jul 07 16:58:09.099639 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:09.099367    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" event={"ID":"b03a0acd-e690-4085-ba36-3e3b148aae0a","Type":"ContainerDied","Data":"e6fc3572e8c326ac8aa09aee664672e5a514febda6ed32b300c70abbf982fe2f"}
Jul 07 16:58:09.135296 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:09.135275    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:58:09.241696 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:09.241668    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03a0acd-e690-4085-ba36-3e3b148aae0a-openshift-service-ca-bundle\") pod \"b03a0acd-e690-4085-ba36-3e3b148aae0a\" (UID: \"b03a0acd-e690-4085-ba36-3e3b148aae0a\") "
Jul 07 16:58:09.241811 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:09.241747    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03a0acd-e690-4085-ba36-3e3b148aae0a-proxy-tls\") pod \"b03a0acd-e690-4085-ba36-3e3b148aae0a\" (UID: \"b03a0acd-e690-4085-ba36-3e3b148aae0a\") "
Jul 07 16:58:09.242084 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:09.242065    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03a0acd-e690-4085-ba36-3e3b148aae0a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b03a0acd-e690-4085-ba36-3e3b148aae0a" (UID: "b03a0acd-e690-4085-ba36-3e3b148aae0a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:09.243673 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:09.243647    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03a0acd-e690-4085-ba36-3e3b148aae0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b03a0acd-e690-4085-ba36-3e3b148aae0a" (UID: "b03a0acd-e690-4085-ba36-3e3b148aae0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:58:09.343232 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:09.343203    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b03a0acd-e690-4085-ba36-3e3b148aae0a-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:58:09.343232 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:09.343228    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03a0acd-e690-4085-ba36-3e3b148aae0a-openshift-service-ca-bundle\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:58:10.105300 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:10.105258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp" event={"ID":"b03a0acd-e690-4085-ba36-3e3b148aae0a","Type":"ContainerDied","Data":"f8051916057aaa646c23752d20c5365ec0f09058173a02cf0b8b51124e80e5a4"}
Jul 07 16:58:10.105300 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:10.105283    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"
Jul 07 16:58:10.105762 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:10.105323    2559 scope.go:117] "RemoveContainer" containerID="e6fc3572e8c326ac8aa09aee664672e5a514febda6ed32b300c70abbf982fe2f"
Jul 07 16:58:10.119608 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:10.119582    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"]
Jul 07 16:58:10.123388 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:10.123362    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ae852-588c44875-cd7jp"]
Jul 07 16:58:11.336091 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:11.336059    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" path="/var/lib/kubelet/pods/b03a0acd-e690-4085-ba36-3e3b148aae0a/volumes"
Jul 07 16:58:12.094556 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:12.094528    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:14.808326 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.808294    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"]
Jul 07 16:58:14.808752 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.808467    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer" containerID="cri-o://62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239" gracePeriod=30
Jul 07 16:58:14.829504 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.829470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:14.971182 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.971131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:14.979135 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.979109    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"]
Jul 07 16:58:14.979444 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.979431    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852"
Jul 07 16:58:14.979444 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.979444    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852"
Jul 07 16:58:14.979562 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.979509    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03a0acd-e690-4085-ba36-3e3b148aae0a" containerName="switch-graph-ae852"
Jul 07 16:58:14.985053 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.983851    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:14.986556 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.986506    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-23477-predictor-serving-cert\""
Jul 07 16:58:14.986696 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.986505    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-23477-kube-rbac-proxy-sar-config\""
Jul 07 16:58:14.992658 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:14.992635    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"]
Jul 07 16:58:15.059152 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.059033    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:15.087433 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.087405    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8048b479-6ad2-43db-bb6f-497cd81bc4f6-proxy-tls\") pod \"success-200-isvc-23477-predictor-69547f9465-pkjgl\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") " pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.087583 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.087446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-23477-predictor-69547f9465-pkjgl\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") " pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.087583 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.087567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz54q\" (UniqueName: \"kubernetes.io/projected/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-api-access-pz54q\") pod \"success-200-isvc-23477-predictor-69547f9465-pkjgl\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") " pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.136949 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.136916    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:15.188836 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.188805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8048b479-6ad2-43db-bb6f-497cd81bc4f6-proxy-tls\") pod \"success-200-isvc-23477-predictor-69547f9465-pkjgl\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") " pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.189026 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.188849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-23477-predictor-69547f9465-pkjgl\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") " pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.189026 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.188920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz54q\" (UniqueName: \"kubernetes.io/projected/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-api-access-pz54q\") pod \"success-200-isvc-23477-predictor-69547f9465-pkjgl\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") " pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.189639 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.189609    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-23477-predictor-69547f9465-pkjgl\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") " pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.191347 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.191326    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8048b479-6ad2-43db-bb6f-497cd81bc4f6-proxy-tls\") pod \"success-200-isvc-23477-predictor-69547f9465-pkjgl\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") " pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.196526 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.196499    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz54q\" (UniqueName: \"kubernetes.io/projected/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-api-access-pz54q\") pod \"success-200-isvc-23477-predictor-69547f9465-pkjgl\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") " pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.238206 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.238177    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:15.297169 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.297143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:15.343480 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.343399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:15.430862 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.430840    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"]
Jul 07 16:58:15.433588 ip-10-0-142-63 kubenswrapper[2559]: W0707 16:58:15.433556    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8048b479_6ad2_43db_bb6f_497cd81bc4f6.slice/crio-b1eb491d38a615557398ffbf7010659334c514cbfe82fae1ad4ecd118e58de32 WatchSource:0}: Error finding container b1eb491d38a615557398ffbf7010659334c514cbfe82fae1ad4ecd118e58de32: Status 404 returned error can't find the container with id b1eb491d38a615557398ffbf7010659334c514cbfe82fae1ad4ecd118e58de32
Jul 07 16:58:15.462936 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:15.462905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:16.134726 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:16.134685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" event={"ID":"8048b479-6ad2-43db-bb6f-497cd81bc4f6","Type":"ContainerStarted","Data":"da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653"}
Jul 07 16:58:16.135136 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:16.134752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" event={"ID":"8048b479-6ad2-43db-bb6f-497cd81bc4f6","Type":"ContainerStarted","Data":"a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e"}
Jul 07 16:58:16.135136 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:16.134775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" event={"ID":"8048b479-6ad2-43db-bb6f-497cd81bc4f6","Type":"ContainerStarted","Data":"b1eb491d38a615557398ffbf7010659334c514cbfe82fae1ad4ecd118e58de32"}
Jul 07 16:58:16.135136 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:16.134971    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:16.151232 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:16.151182    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" podStartSLOduration=2.151166545 podStartE2EDuration="2.151166545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:58:16.148965667 +0000 UTC m=+634.412502309" watchObservedRunningTime="2026-07-07 16:58:16.151166545 +0000 UTC m=+634.414703177"
Jul 07 16:58:16.979316 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:16.979271    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 16:58:17.093269 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:17.093228    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:17.139281 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:17.139250    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:17.140715 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:17.140687    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 16:58:18.143419 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:18.143380    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 16:58:22.092906 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:22.092866    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:23.148410 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:23.148380    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:58:23.148836 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:23.148809    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 16:58:26.980889 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:26.980856    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 16:58:27.092596 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:27.092556    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:27.092765 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:27.092660    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:32.092484 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:32.092437    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:33.149105 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:33.149069    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 16:58:37.092155 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:37.092108    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:39.218102 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:39.218063    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:39.392244 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:39.392215    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:42.092313 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:42.092279    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:43.149346 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:43.149306    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 16:58:44.951961 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:44.951939    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:45.008439 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.008411    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115662c5-a931-43d7-b818-d9d71524937d-openshift-service-ca-bundle\") pod \"115662c5-a931-43d7-b818-d9d71524937d\" (UID: \"115662c5-a931-43d7-b818-d9d71524937d\") "
Jul 07 16:58:45.008553 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.008465    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115662c5-a931-43d7-b818-d9d71524937d-proxy-tls\") pod \"115662c5-a931-43d7-b818-d9d71524937d\" (UID: \"115662c5-a931-43d7-b818-d9d71524937d\") "
Jul 07 16:58:45.008767 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.008747    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115662c5-a931-43d7-b818-d9d71524937d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "115662c5-a931-43d7-b818-d9d71524937d" (UID: "115662c5-a931-43d7-b818-d9d71524937d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:45.010409 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.010383    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115662c5-a931-43d7-b818-d9d71524937d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "115662c5-a931-43d7-b818-d9d71524937d" (UID: "115662c5-a931-43d7-b818-d9d71524937d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:58:45.109142 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.109080    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115662c5-a931-43d7-b818-d9d71524937d-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:58:45.109142 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.109104    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/115662c5-a931-43d7-b818-d9d71524937d-openshift-service-ca-bundle\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 16:58:45.266445 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.266415    2559 generic.go:356] "Generic (PLEG): container finished" podID="115662c5-a931-43d7-b818-d9d71524937d" containerID="62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239" exitCode=0
Jul 07 16:58:45.266673 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.266462    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"
Jul 07 16:58:45.266673 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.266502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" event={"ID":"115662c5-a931-43d7-b818-d9d71524937d","Type":"ContainerDied","Data":"62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239"}
Jul 07 16:58:45.266673 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.266560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz" event={"ID":"115662c5-a931-43d7-b818-d9d71524937d","Type":"ContainerDied","Data":"a9052ae71cfded379c8a8ae2b4fb4ac8a3b9240fa31708666ba1672bb214ffec"}
Jul 07 16:58:45.266673 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.266588    2559 scope.go:117] "RemoveContainer" containerID="62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239"
Jul 07 16:58:45.274880 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.274864    2559 scope.go:117] "RemoveContainer" containerID="62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239"
Jul 07 16:58:45.275137 ip-10-0-142-63 kubenswrapper[2559]: E0707 16:58:45.275119    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239\": container with ID starting with 62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239 not found: ID does not exist" containerID="62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239"
Jul 07 16:58:45.275180 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.275146    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239"} err="failed to get container status \"62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239\": rpc error: code = NotFound desc = could not find container \"62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239\": container with ID starting with 62a1e4f9c9145f9ea63b8a176239241036f9a9657f177cdec889dc33ed1b7239 not found: ID does not exist"
Jul 07 16:58:45.287353 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.287329    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"]
Jul 07 16:58:45.290963 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.290943    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-565d7b4557-w6klz"]
Jul 07 16:58:45.340294 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:45.340265    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="115662c5-a931-43d7-b818-d9d71524937d" path="/var/lib/kubelet/pods/115662c5-a931-43d7-b818-d9d71524937d/volumes"
Jul 07 16:58:52.086356 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:52.086317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:52.186493 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:52.186458    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:53.149576 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:58:53.149540    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 16:59:03.149905 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:59:03.149871    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 16:59:14.996823 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:59:14.996784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 16:59:15.484910 ip-10-0-142-63 kubenswrapper[2559]: I0707 16:59:15.484873    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:00:27.967324 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:00:27.967289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:00:28.068935 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:00:28.068904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:02:59.602705 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:02:59.602647    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:02:59.688240 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:02:59.688209    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:04:31.477793 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:04:31.477711    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:06:48.168751 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:48.168718    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:06:51.116504 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:51.116469    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:06:53.853176 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:53.853146    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:06:53.997858 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:53.997812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:06:54.045923 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.045881    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"]
Jul 07 17:06:54.046671 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.046644    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer"
Jul 07 17:06:54.046840 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.046674    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer"
Jul 07 17:06:54.046840 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.046783    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="115662c5-a931-43d7-b818-d9d71524937d" containerName="model-chainer"
Jul 07 17:06:54.050286 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.050265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.053576 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.053554    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2614e-predictor-serving-cert\""
Jul 07 17:06:54.054502 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.054474    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2614e-kube-rbac-proxy-sar-config\""
Jul 07 17:06:54.061311 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.061289    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"]
Jul 07 17:06:54.065706 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.065684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb560f9e-c359-4ec8-8482-3f575188e780-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.065797 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.065722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb560f9e-c359-4ec8-8482-3f575188e780-proxy-tls\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.065797 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.065768    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566cv\" (UniqueName: \"kubernetes.io/projected/bb560f9e-c359-4ec8-8482-3f575188e780-kube-api-access-566cv\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.074345 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.074322    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"]
Jul 07 17:06:54.074763 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.074732    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container" containerID="cri-o://d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074" gracePeriod=30
Jul 07 17:06:54.075049 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.074773    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kube-rbac-proxy" containerID="cri-o://45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672" gracePeriod=30
Jul 07 17:06:54.144518 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.144445    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:06:54.166526 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.166494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb560f9e-c359-4ec8-8482-3f575188e780-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.166681 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.166551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb560f9e-c359-4ec8-8482-3f575188e780-proxy-tls\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.166681 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.166601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566cv\" (UniqueName: \"kubernetes.io/projected/bb560f9e-c359-4ec8-8482-3f575188e780-kube-api-access-566cv\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.166811 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:06:54.166703    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-2614e-predictor-serving-cert: secret "success-200-isvc-2614e-predictor-serving-cert" not found
Jul 07 17:06:54.166811 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:06:54.166777    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb560f9e-c359-4ec8-8482-3f575188e780-proxy-tls podName:bb560f9e-c359-4ec8-8482-3f575188e780 nodeName:}" failed. No retries permitted until 2026-07-07 17:06:54.666754342 +0000 UTC m=+1152.930290969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bb560f9e-c359-4ec8-8482-3f575188e780-proxy-tls") pod "success-200-isvc-2614e-predictor-586c86c5f-kmrfh" (UID: "bb560f9e-c359-4ec8-8482-3f575188e780") : secret "success-200-isvc-2614e-predictor-serving-cert" not found
Jul 07 17:06:54.167231 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.167210    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb560f9e-c359-4ec8-8482-3f575188e780-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.181669 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.181641    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566cv\" (UniqueName: \"kubernetes.io/projected/bb560f9e-c359-4ec8-8482-3f575188e780-kube-api-access-566cv\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.231517 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.231493    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:06:54.310547 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.310515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:06:54.365577 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.365552    2559 generic.go:356] "Generic (PLEG): container finished" podID="c73352ca-099a-4c85-b42f-af052a96caa1" containerID="45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672" exitCode=2
Jul 07 17:06:54.365723 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.365631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" event={"ID":"c73352ca-099a-4c85-b42f-af052a96caa1","Type":"ContainerDied","Data":"45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672"}
Jul 07 17:06:54.669843 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.669804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb560f9e-c359-4ec8-8482-3f575188e780-proxy-tls\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.672239 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.672219    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb560f9e-c359-4ec8-8482-3f575188e780-proxy-tls\") pod \"success-200-isvc-2614e-predictor-586c86c5f-kmrfh\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") " pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:54.854183 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.854153    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:06:54.963931 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:54.963841    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:55.100043 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:55.099963    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"]
Jul 07 17:06:55.102439 ip-10-0-142-63 kubenswrapper[2559]: W0707 17:06:55.102416    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb560f9e_c359_4ec8_8482_3f575188e780.slice/crio-61d7c58bc7658562ad4c1e9dfb09d21960ad75cd6c324da1914eefe733dd8205 WatchSource:0}: Error finding container 61d7c58bc7658562ad4c1e9dfb09d21960ad75cd6c324da1914eefe733dd8205: Status 404 returned error can't find the container with id 61d7c58bc7658562ad4c1e9dfb09d21960ad75cd6c324da1914eefe733dd8205
Jul 07 17:06:55.374536 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:55.374499    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" event={"ID":"bb560f9e-c359-4ec8-8482-3f575188e780","Type":"ContainerStarted","Data":"606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af"}
Jul 07 17:06:55.374720 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:55.374581    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:55.374720 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:55.374607    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:06:55.374720 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:55.374629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" event={"ID":"bb560f9e-c359-4ec8-8482-3f575188e780","Type":"ContainerStarted","Data":"730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464"}
Jul 07 17:06:55.374720 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:55.374648    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" event={"ID":"bb560f9e-c359-4ec8-8482-3f575188e780","Type":"ContainerStarted","Data":"61d7c58bc7658562ad4c1e9dfb09d21960ad75cd6c324da1914eefe733dd8205"}
Jul 07 17:06:55.376184 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:55.376151    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 07 17:06:55.402820 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:55.402770    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podStartSLOduration=1.4027576800000001 podStartE2EDuration="1.40275768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:06:55.401210312 +0000 UTC m=+1153.664746972" watchObservedRunningTime="2026-07-07 17:06:55.40275768 +0000 UTC m=+1153.666294311"
Jul 07 17:06:56.379065 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:56.379027    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 07 17:06:56.975630 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:56.975592    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 07 17:06:56.979926 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:56.979900    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 17:06:57.127747 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.127726    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 17:06:57.189462 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.189403    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73352ca-099a-4c85-b42f-af052a96caa1-kube-rbac-proxy-sar-config\") pod \"c73352ca-099a-4c85-b42f-af052a96caa1\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") "
Jul 07 17:06:57.189462 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.189435    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84km5\" (UniqueName: \"kubernetes.io/projected/c73352ca-099a-4c85-b42f-af052a96caa1-kube-api-access-84km5\") pod \"c73352ca-099a-4c85-b42f-af052a96caa1\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") "
Jul 07 17:06:57.189609 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.189486    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73352ca-099a-4c85-b42f-af052a96caa1-proxy-tls\") pod \"c73352ca-099a-4c85-b42f-af052a96caa1\" (UID: \"c73352ca-099a-4c85-b42f-af052a96caa1\") "
Jul 07 17:06:57.189801 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.189781    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73352ca-099a-4c85-b42f-af052a96caa1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c73352ca-099a-4c85-b42f-af052a96caa1" (UID: "c73352ca-099a-4c85-b42f-af052a96caa1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:06:57.191488 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.191464    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73352ca-099a-4c85-b42f-af052a96caa1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c73352ca-099a-4c85-b42f-af052a96caa1" (UID: "c73352ca-099a-4c85-b42f-af052a96caa1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:06:57.191585 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.191514    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73352ca-099a-4c85-b42f-af052a96caa1-kube-api-access-84km5" (OuterVolumeSpecName: "kube-api-access-84km5") pod "c73352ca-099a-4c85-b42f-af052a96caa1" (UID: "c73352ca-099a-4c85-b42f-af052a96caa1"). InnerVolumeSpecName "kube-api-access-84km5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:06:57.290210 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.290181    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c73352ca-099a-4c85-b42f-af052a96caa1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:06:57.290210 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.290205    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84km5\" (UniqueName: \"kubernetes.io/projected/c73352ca-099a-4c85-b42f-af052a96caa1-kube-api-access-84km5\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:06:57.290210 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.290215    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c73352ca-099a-4c85-b42f-af052a96caa1-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:06:57.384222 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.384189    2559 generic.go:356] "Generic (PLEG): container finished" podID="c73352ca-099a-4c85-b42f-af052a96caa1" containerID="d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074" exitCode=0
Jul 07 17:06:57.384539 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.384238    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"
Jul 07 17:06:57.384539 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.384233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" event={"ID":"c73352ca-099a-4c85-b42f-af052a96caa1","Type":"ContainerDied","Data":"d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074"}
Jul 07 17:06:57.384539 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.384368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh" event={"ID":"c73352ca-099a-4c85-b42f-af052a96caa1","Type":"ContainerDied","Data":"ca32d8167ae470fa2dd29f9042727cb7808ef5aed2042e167c193ff25dd8f7c0"}
Jul 07 17:06:57.384539 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.384403    2559 scope.go:117] "RemoveContainer" containerID="45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672"
Jul 07 17:06:57.392799 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.392778    2559 scope.go:117] "RemoveContainer" containerID="d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074"
Jul 07 17:06:57.400955 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.400933    2559 scope.go:117] "RemoveContainer" containerID="45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672"
Jul 07 17:06:57.401257 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:06:57.401237    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672\": container with ID starting with 45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672 not found: ID does not exist" containerID="45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672"
Jul 07 17:06:57.401305 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.401265    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672"} err="failed to get container status \"45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672\": rpc error: code = NotFound desc = could not find container \"45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672\": container with ID starting with 45aa13580db6cf256eec5da870f5206d6e36fab344f8c42ab527882f04a7c672 not found: ID does not exist"
Jul 07 17:06:57.401305 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.401282    2559 scope.go:117] "RemoveContainer" containerID="d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074"
Jul 07 17:06:57.401536 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:06:57.401518    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074\": container with ID starting with d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074 not found: ID does not exist" containerID="d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074"
Jul 07 17:06:57.401589 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.401544    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074"} err="failed to get container status \"d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074\": rpc error: code = NotFound desc = could not find container \"d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074\": container with ID starting with d331787412ef195ff06d19ba59812fa9e2d375a76ad8854d618720ea2d4a4074 not found: ID does not exist"
Jul 07 17:06:57.404442 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.404412    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"]
Jul 07 17:06:57.410168 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:57.410143    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb9ec-predictor-6966f7898f-2r7kh"]
Jul 07 17:06:59.335997 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:06:59.335949    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" path="/var/lib/kubelet/pods/c73352ca-099a-4c85-b42f-af052a96caa1/volumes"
Jul 07 17:07:01.383776 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:01.383750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:07:01.384253 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:01.384231    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 07 17:07:11.384682 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:11.384619    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 07 17:07:21.384534 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:21.384496    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 07 17:07:29.712841 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.712803    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:29.866003 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.865956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:29.873152 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.873126    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"]
Jul 07 17:07:29.873354 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.873333    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container" containerID="cri-o://a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e" gracePeriod=30
Jul 07 17:07:29.873431 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.873375    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kube-rbac-proxy" containerID="cri-o://da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653" gracePeriod=30
Jul 07 17:07:29.891770 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.891735    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"]
Jul 07 17:07:29.892150 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.892130    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container"
Jul 07 17:07:29.892150 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.892148    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container"
Jul 07 17:07:29.892323 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.892161    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kube-rbac-proxy"
Jul 07 17:07:29.892323 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.892169    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kube-rbac-proxy"
Jul 07 17:07:29.892323 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.892266    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kserve-container"
Jul 07 17:07:29.892323 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.892277    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73352ca-099a-4c85-b42f-af052a96caa1" containerName="kube-rbac-proxy"
Jul 07 17:07:29.897282 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.897261    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:29.899885 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.899848    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8411f-predictor-serving-cert\""
Jul 07 17:07:29.900011 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.899900    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8411f-kube-rbac-proxy-sar-config\""
Jul 07 17:07:29.913149 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.913106    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"]
Jul 07 17:07:29.960413 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.960384    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601ec390-60cf-47aa-817c-c708ace44d65-proxy-tls\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:29.960545 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.960434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601ec390-60cf-47aa-817c-c708ace44d65-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:29.960545 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.960520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stfcx\" (UniqueName: \"kubernetes.io/projected/601ec390-60cf-47aa-817c-c708ace44d65-kube-api-access-stfcx\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:29.971258 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:29.971180    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:30.061699 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.061659    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601ec390-60cf-47aa-817c-c708ace44d65-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:30.061866 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.061738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stfcx\" (UniqueName: \"kubernetes.io/projected/601ec390-60cf-47aa-817c-c708ace44d65-kube-api-access-stfcx\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:30.061866 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.061808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601ec390-60cf-47aa-817c-c708ace44d65-proxy-tls\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:30.062030 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:07:30.061925    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8411f-predictor-serving-cert: secret "success-200-isvc-8411f-predictor-serving-cert" not found
Jul 07 17:07:30.062030 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:07:30.062023    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601ec390-60cf-47aa-817c-c708ace44d65-proxy-tls podName:601ec390-60cf-47aa-817c-c708ace44d65 nodeName:}" failed. No retries permitted until 2026-07-07 17:07:30.562000765 +0000 UTC m=+1188.825537378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/601ec390-60cf-47aa-817c-c708ace44d65-proxy-tls") pod "success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" (UID: "601ec390-60cf-47aa-817c-c708ace44d65") : secret "success-200-isvc-8411f-predictor-serving-cert" not found
Jul 07 17:07:30.062448 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.062426    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601ec390-60cf-47aa-817c-c708ace44d65-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:30.085702 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.085672    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stfcx\" (UniqueName: \"kubernetes.io/projected/601ec390-60cf-47aa-817c-c708ace44d65-kube-api-access-stfcx\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:30.097290 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.097260    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:30.189488 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.189434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:30.383367 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.383325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:30.489804 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.489771    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:30.530714 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.530685    2559 generic.go:356] "Generic (PLEG): container finished" podID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerID="da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653" exitCode=2
Jul 07 17:07:30.530848 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.530745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" event={"ID":"8048b479-6ad2-43db-bb6f-497cd81bc4f6","Type":"ContainerDied","Data":"da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653"}
Jul 07 17:07:30.567239 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.567213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601ec390-60cf-47aa-817c-c708ace44d65-proxy-tls\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:30.569515 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.569493    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601ec390-60cf-47aa-817c-c708ace44d65-proxy-tls\") pod \"success-200-isvc-8411f-predictor-55547cb9c5-5bbx6\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") " pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:30.808016 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.807970    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:30.945412 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:30.945380    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"]
Jul 07 17:07:30.950525 ip-10-0-142-63 kubenswrapper[2559]: W0707 17:07:30.950495    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601ec390_60cf_47aa_817c_c708ace44d65.slice/crio-78d306f3dc0c5f7d67fd7902468cc856a01240cf3dee77be355b0862b63fab1f WatchSource:0}: Error finding container 78d306f3dc0c5f7d67fd7902468cc856a01240cf3dee77be355b0862b63fab1f: Status 404 returned error can't find the container with id 78d306f3dc0c5f7d67fd7902468cc856a01240cf3dee77be355b0862b63fab1f
Jul 07 17:07:31.384202 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:31.384126    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 07 17:07:31.537141 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:31.537108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" event={"ID":"601ec390-60cf-47aa-817c-c708ace44d65","Type":"ContainerStarted","Data":"22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984"}
Jul 07 17:07:31.537290 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:31.537181    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" event={"ID":"601ec390-60cf-47aa-817c-c708ace44d65","Type":"ContainerStarted","Data":"9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275"}
Jul 07 17:07:31.537290 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:31.537204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" event={"ID":"601ec390-60cf-47aa-817c-c708ace44d65","Type":"ContainerStarted","Data":"78d306f3dc0c5f7d67fd7902468cc856a01240cf3dee77be355b0862b63fab1f"}
Jul 07 17:07:31.537290 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:31.537226    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:31.555413 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:31.555357    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" podStartSLOduration=2.555341577 podStartE2EDuration="2.555341577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:07:31.554167814 +0000 UTC m=+1189.817704464" watchObservedRunningTime="2026-07-07 17:07:31.555341577 +0000 UTC m=+1189.818878277"
Jul 07 17:07:32.541993 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:32.541937    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:32.543386 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:32.543362    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 07 17:07:33.128051 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.128026    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 17:07:33.291507 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.291477    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz54q\" (UniqueName: \"kubernetes.io/projected/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-api-access-pz54q\") pod \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") "
Jul 07 17:07:33.291659 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.291538    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8048b479-6ad2-43db-bb6f-497cd81bc4f6-proxy-tls\") pod \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") "
Jul 07 17:07:33.291659 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.291566    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-rbac-proxy-sar-config\") pod \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\" (UID: \"8048b479-6ad2-43db-bb6f-497cd81bc4f6\") "
Jul 07 17:07:33.291897 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.291878    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8048b479-6ad2-43db-bb6f-497cd81bc4f6" (UID: "8048b479-6ad2-43db-bb6f-497cd81bc4f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:07:33.293591 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.293564    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-api-access-pz54q" (OuterVolumeSpecName: "kube-api-access-pz54q") pod "8048b479-6ad2-43db-bb6f-497cd81bc4f6" (UID: "8048b479-6ad2-43db-bb6f-497cd81bc4f6"). InnerVolumeSpecName "kube-api-access-pz54q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:07:33.293669 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.293611    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8048b479-6ad2-43db-bb6f-497cd81bc4f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8048b479-6ad2-43db-bb6f-497cd81bc4f6" (UID: "8048b479-6ad2-43db-bb6f-497cd81bc4f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:07:33.392382 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.392353    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz54q\" (UniqueName: \"kubernetes.io/projected/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-api-access-pz54q\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:07:33.392382 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.392380    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8048b479-6ad2-43db-bb6f-497cd81bc4f6-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:07:33.392539 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.392391    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8048b479-6ad2-43db-bb6f-497cd81bc4f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:07:33.550174 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.550089    2559 generic.go:356] "Generic (PLEG): container finished" podID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerID="a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e" exitCode=0
Jul 07 17:07:33.550174 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.550159    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"
Jul 07 17:07:33.550174 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.550169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" event={"ID":"8048b479-6ad2-43db-bb6f-497cd81bc4f6","Type":"ContainerDied","Data":"a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e"}
Jul 07 17:07:33.550766 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.550236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl" event={"ID":"8048b479-6ad2-43db-bb6f-497cd81bc4f6","Type":"ContainerDied","Data":"b1eb491d38a615557398ffbf7010659334c514cbfe82fae1ad4ecd118e58de32"}
Jul 07 17:07:33.550766 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.550263    2559 scope.go:117] "RemoveContainer" containerID="da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653"
Jul 07 17:07:33.550872 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.550815    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 07 17:07:33.558139 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.558078    2559 scope.go:117] "RemoveContainer" containerID="a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e"
Jul 07 17:07:33.565198 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.565180    2559 scope.go:117] "RemoveContainer" containerID="da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653"
Jul 07 17:07:33.565441 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:07:33.565421    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653\": container with ID starting with da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653 not found: ID does not exist" containerID="da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653"
Jul 07 17:07:33.565482 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.565450    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653"} err="failed to get container status \"da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653\": rpc error: code = NotFound desc = could not find container \"da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653\": container with ID starting with da3ff67befaa3dfc9a12f58f9ca18573b2daf7a4671de68e38b73295f0bc2653 not found: ID does not exist"
Jul 07 17:07:33.565482 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.565466    2559 scope.go:117] "RemoveContainer" containerID="a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e"
Jul 07 17:07:33.565704 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:07:33.565687    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e\": container with ID starting with a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e not found: ID does not exist" containerID="a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e"
Jul 07 17:07:33.565758 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.565710    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e"} err="failed to get container status \"a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e\": rpc error: code = NotFound desc = could not find container \"a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e\": container with ID starting with a972d33b0ef34d5ded04413eea96a67319851c383c3b77752feb06ec1732428e not found: ID does not exist"
Jul 07 17:07:33.577382 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.577363    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"]
Jul 07 17:07:33.583952 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:33.583932    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-23477-predictor-69547f9465-pkjgl"]
Jul 07 17:07:35.335899 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:35.335864    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" path="/var/lib/kubelet/pods/8048b479-6ad2-43db-bb6f-497cd81bc4f6/volumes"
Jul 07 17:07:38.554422 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:38.554390    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:07:38.555052 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:38.555019    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 07 17:07:41.385857 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:41.385825    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:07:48.555084 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:48.555046    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 07 17:07:54.076742 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:54.076708    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:54.271942 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:54.271899    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:58.555108 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:07:58.555064    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 07 17:08:04.189587 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.189506    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:04.267902 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.267862    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:04.295208 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.295176    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"]
Jul 07 17:08:04.295458 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.295429    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container" containerID="cri-o://730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464" gracePeriod=30
Jul 07 17:08:04.295570 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.295477    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kube-rbac-proxy" containerID="cri-o://606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af" gracePeriod=30
Jul 07 17:08:04.307007 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.306962    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"]
Jul 07 17:08:04.307395 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.307379    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container"
Jul 07 17:08:04.307466 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.307399    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container"
Jul 07 17:08:04.307466 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.307429    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kube-rbac-proxy"
Jul 07 17:08:04.307466 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.307440    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kube-rbac-proxy"
Jul 07 17:08:04.307618 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.307528    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kube-rbac-proxy"
Jul 07 17:08:04.307618 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.307543    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8048b479-6ad2-43db-bb6f-497cd81bc4f6" containerName="kserve-container"
Jul 07 17:08:04.310617 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.310597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.313174 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.313152    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f8ada-predictor-serving-cert\""
Jul 07 17:08:04.313254 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.313152    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f8ada-kube-rbac-proxy-sar-config\""
Jul 07 17:08:04.323257 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.323016    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jlv\" (UniqueName: \"kubernetes.io/projected/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-api-access-j5jlv\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.323257 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.323071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.323257 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.323158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7bd633f-19f0-4ca2-8394-c2f81f97526f-proxy-tls\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.325392 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.325361    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"]
Jul 07 17:08:04.369089 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.369050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:04.423985 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.423945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.424131 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.424103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7bd633f-19f0-4ca2-8394-c2f81f97526f-proxy-tls\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.424350 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:08:04.424331    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-serving-cert: secret "success-200-isvc-f8ada-predictor-serving-cert" not found
Jul 07 17:08:04.424421 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:08:04.424400    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7bd633f-19f0-4ca2-8394-c2f81f97526f-proxy-tls podName:e7bd633f-19f0-4ca2-8394-c2f81f97526f nodeName:}" failed. No retries permitted until 2026-07-07 17:08:04.924378363 +0000 UTC m=+1223.187914978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e7bd633f-19f0-4ca2-8394-c2f81f97526f-proxy-tls") pod "success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" (UID: "e7bd633f-19f0-4ca2-8394-c2f81f97526f") : secret "success-200-isvc-f8ada-predictor-serving-cert" not found
Jul 07 17:08:04.424656 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.424628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jlv\" (UniqueName: \"kubernetes.io/projected/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-api-access-j5jlv\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.424751 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.424735    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.435319 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.435291    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jlv\" (UniqueName: \"kubernetes.io/projected/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-api-access-j5jlv\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.442660 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.442592    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:04.616421 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.616383    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:04.686407 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.686378    2559 generic.go:356] "Generic (PLEG): container finished" podID="bb560f9e-c359-4ec8-8482-3f575188e780" containerID="606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af" exitCode=2
Jul 07 17:08:04.686543 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.686451    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" event={"ID":"bb560f9e-c359-4ec8-8482-3f575188e780","Type":"ContainerDied","Data":"606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af"}
Jul 07 17:08:04.716730 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.716633    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:04.928501 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.928469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7bd633f-19f0-4ca2-8394-c2f81f97526f-proxy-tls\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:04.930815 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:04.930793    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7bd633f-19f0-4ca2-8394-c2f81f97526f-proxy-tls\") pod \"success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") " pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:05.228040 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:05.228005    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:05.355942 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:05.355919    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"]
Jul 07 17:08:05.357901 ip-10-0-142-63 kubenswrapper[2559]: W0707 17:08:05.357875    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bd633f_19f0_4ca2_8394_c2f81f97526f.slice/crio-53a4620fa5f9130c1e3c9d1a92083f051c111616c084177f84cd940471d2bd41 WatchSource:0}: Error finding container 53a4620fa5f9130c1e3c9d1a92083f051c111616c084177f84cd940471d2bd41: Status 404 returned error can't find the container with id 53a4620fa5f9130c1e3c9d1a92083f051c111616c084177f84cd940471d2bd41
Jul 07 17:08:05.693293 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:05.693254    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" event={"ID":"e7bd633f-19f0-4ca2-8394-c2f81f97526f","Type":"ContainerStarted","Data":"7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b"}
Jul 07 17:08:05.693479 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:05.693320    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" event={"ID":"e7bd633f-19f0-4ca2-8394-c2f81f97526f","Type":"ContainerStarted","Data":"558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359"}
Jul 07 17:08:05.693479 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:05.693347    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" event={"ID":"e7bd633f-19f0-4ca2-8394-c2f81f97526f","Type":"ContainerStarted","Data":"53a4620fa5f9130c1e3c9d1a92083f051c111616c084177f84cd940471d2bd41"}
Jul 07 17:08:05.693479 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:05.693383    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:05.710314 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:05.710099    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" podStartSLOduration=1.710079827 podStartE2EDuration="1.710079827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:08:05.708511263 +0000 UTC m=+1223.972047915" watchObservedRunningTime="2026-07-07 17:08:05.710079827 +0000 UTC m=+1223.973616459"
Jul 07 17:08:06.380198 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:06.380161    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 07 17:08:06.698273 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:06.698179    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:06.699542 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:06.699519    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 07 17:08:07.348561 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.348538    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:08:07.445841 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.445769    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-566cv\" (UniqueName: \"kubernetes.io/projected/bb560f9e-c359-4ec8-8482-3f575188e780-kube-api-access-566cv\") pod \"bb560f9e-c359-4ec8-8482-3f575188e780\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") "
Jul 07 17:08:07.445841 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.445822    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb560f9e-c359-4ec8-8482-3f575188e780-kube-rbac-proxy-sar-config\") pod \"bb560f9e-c359-4ec8-8482-3f575188e780\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") "
Jul 07 17:08:07.446287 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.445850    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb560f9e-c359-4ec8-8482-3f575188e780-proxy-tls\") pod \"bb560f9e-c359-4ec8-8482-3f575188e780\" (UID: \"bb560f9e-c359-4ec8-8482-3f575188e780\") "
Jul 07 17:08:07.446287 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.446175    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb560f9e-c359-4ec8-8482-3f575188e780-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bb560f9e-c359-4ec8-8482-3f575188e780" (UID: "bb560f9e-c359-4ec8-8482-3f575188e780"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:08:07.447869 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.447847    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb560f9e-c359-4ec8-8482-3f575188e780-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb560f9e-c359-4ec8-8482-3f575188e780" (UID: "bb560f9e-c359-4ec8-8482-3f575188e780"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:08:07.447950 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.447932    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb560f9e-c359-4ec8-8482-3f575188e780-kube-api-access-566cv" (OuterVolumeSpecName: "kube-api-access-566cv") pod "bb560f9e-c359-4ec8-8482-3f575188e780" (UID: "bb560f9e-c359-4ec8-8482-3f575188e780"). InnerVolumeSpecName "kube-api-access-566cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:08:07.547227 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.547200    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb560f9e-c359-4ec8-8482-3f575188e780-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:08:07.547227 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.547222    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-566cv\" (UniqueName: \"kubernetes.io/projected/bb560f9e-c359-4ec8-8482-3f575188e780-kube-api-access-566cv\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:08:07.547381 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.547233    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb560f9e-c359-4ec8-8482-3f575188e780-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:08:07.709390 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.709290    2559 generic.go:356] "Generic (PLEG): container finished" podID="bb560f9e-c359-4ec8-8482-3f575188e780" containerID="730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464" exitCode=0
Jul 07 17:08:07.709390 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.709337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" event={"ID":"bb560f9e-c359-4ec8-8482-3f575188e780","Type":"ContainerDied","Data":"730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464"}
Jul 07 17:08:07.709390 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.709361    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"
Jul 07 17:08:07.709660 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.709418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh" event={"ID":"bb560f9e-c359-4ec8-8482-3f575188e780","Type":"ContainerDied","Data":"61d7c58bc7658562ad4c1e9dfb09d21960ad75cd6c324da1914eefe733dd8205"}
Jul 07 17:08:07.709660 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.709434    2559 scope.go:117] "RemoveContainer" containerID="606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af"
Jul 07 17:08:07.710187 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.710153    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 07 17:08:07.719499 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.719482    2559 scope.go:117] "RemoveContainer" containerID="730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464"
Jul 07 17:08:07.726695 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.726665    2559 scope.go:117] "RemoveContainer" containerID="606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af"
Jul 07 17:08:07.726934 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:08:07.726915    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af\": container with ID starting with 606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af not found: ID does not exist" containerID="606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af"
Jul 07 17:08:07.727101 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.726942    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af"} err="failed to get container status \"606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af\": rpc error: code = NotFound desc = could not find container \"606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af\": container with ID starting with 606a83ea79ae652d1f28520a5102336b0e2e14847b50743c1758182b8b7089af not found: ID does not exist"
Jul 07 17:08:07.727101 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.726958    2559 scope.go:117] "RemoveContainer" containerID="730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464"
Jul 07 17:08:07.727354 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:08:07.727334    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464\": container with ID starting with 730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464 not found: ID does not exist" containerID="730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464"
Jul 07 17:08:07.727415 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.727362    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464"} err="failed to get container status \"730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464\": rpc error: code = NotFound desc = could not find container \"730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464\": container with ID starting with 730bf66481b27e995344e060e8bcf42253f6bba44453cd3c27d34d2d7ee6c464 not found: ID does not exist"
Jul 07 17:08:07.729966 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.729941    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"]
Jul 07 17:08:07.736228 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:07.736202    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2614e-predictor-586c86c5f-kmrfh"]
Jul 07 17:08:08.555566 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:08.555522    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 07 17:08:09.336868 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:09.336834    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" path="/var/lib/kubelet/pods/bb560f9e-c359-4ec8-8482-3f575188e780/volumes"
Jul 07 17:08:12.714890 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:12.714856    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:12.715477 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:12.715454    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 07 17:08:18.556023 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:18.555989    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:08:22.715497 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:22.715456    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 07 17:08:28.380492 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:28.380455    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:28.481105 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:28.481066    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:29.858763 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:29.858717    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:30.109726 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:30.109640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:32.715674 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:32.715623    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 07 17:08:39.969268 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:39.969220    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:40.058252 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.058207    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"]
Jul 07 17:08:40.058616 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.058580    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container" containerID="cri-o://9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275" gracePeriod=30
Jul 07 17:08:40.058762 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.058650    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kube-rbac-proxy" containerID="cri-o://22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984" gracePeriod=30
Jul 07 17:08:40.072124 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.072095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:40.084276 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.084252    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"]
Jul 07 17:08:40.084634 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.084618    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container"
Jul 07 17:08:40.084711 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.084637    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container"
Jul 07 17:08:40.084711 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.084664    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kube-rbac-proxy"
Jul 07 17:08:40.084711 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.084673    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kube-rbac-proxy"
Jul 07 17:08:40.084862 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.084752    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kube-rbac-proxy"
Jul 07 17:08:40.084862 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.084766    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb560f9e-c359-4ec8-8482-3f575188e780" containerName="kserve-container"
Jul 07 17:08:40.088012 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.087994    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.089946 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.089928    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3653b-predictor-serving-cert\""
Jul 07 17:08:40.090046 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.089952    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3653b-kube-rbac-proxy-sar-config\""
Jul 07 17:08:40.096142 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.096122    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"]
Jul 07 17:08:40.203240 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.203198    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b171b25-3089-445b-a7a5-65a5c23198be-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3653b-predictor-849b9cdd5-4p4pj\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") " pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.203419 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.203254    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw7sl\" (UniqueName: \"kubernetes.io/projected/4b171b25-3089-445b-a7a5-65a5c23198be-kube-api-access-pw7sl\") pod \"success-200-isvc-3653b-predictor-849b9cdd5-4p4pj\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") " pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.203419 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.203371    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b171b25-3089-445b-a7a5-65a5c23198be-proxy-tls\") pod \"success-200-isvc-3653b-predictor-849b9cdd5-4p4pj\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") " pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.304177 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.304144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b171b25-3089-445b-a7a5-65a5c23198be-proxy-tls\") pod \"success-200-isvc-3653b-predictor-849b9cdd5-4p4pj\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") " pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.304344 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.304191    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b171b25-3089-445b-a7a5-65a5c23198be-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3653b-predictor-849b9cdd5-4p4pj\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") " pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.304344 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.304217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7sl\" (UniqueName: \"kubernetes.io/projected/4b171b25-3089-445b-a7a5-65a5c23198be-kube-api-access-pw7sl\") pod \"success-200-isvc-3653b-predictor-849b9cdd5-4p4pj\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") " pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.304828 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.304804    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b171b25-3089-445b-a7a5-65a5c23198be-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3653b-predictor-849b9cdd5-4p4pj\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") " pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.306492 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.306467    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b171b25-3089-445b-a7a5-65a5c23198be-proxy-tls\") pod \"success-200-isvc-3653b-predictor-849b9cdd5-4p4pj\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") " pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.311774 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.311749    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7sl\" (UniqueName: \"kubernetes.io/projected/4b171b25-3089-445b-a7a5-65a5c23198be-kube-api-access-pw7sl\") pod \"success-200-isvc-3653b-predictor-849b9cdd5-4p4pj\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") " pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.399706 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.399682    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.508809 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.508777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:40.523701 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.523684    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"]
Jul 07 17:08:40.525890 ip-10-0-142-63 kubenswrapper[2559]: W0707 17:08:40.525868    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b171b25_3089_445b_a7a5_65a5c23198be.slice/crio-e26731a9e8686aaf6281ccbd537e182e2782c6b7c7e32cc58cc55d1345d2891b WatchSource:0}: Error finding container e26731a9e8686aaf6281ccbd537e182e2782c6b7c7e32cc58cc55d1345d2891b: Status 404 returned error can't find the container with id e26731a9e8686aaf6281ccbd537e182e2782c6b7c7e32cc58cc55d1345d2891b
Jul 07 17:08:40.609270 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.609206    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:08:40.857181 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.857076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" event={"ID":"4b171b25-3089-445b-a7a5-65a5c23198be","Type":"ContainerStarted","Data":"fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843"}
Jul 07 17:08:40.857181 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.857156    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.857564 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.857184    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" event={"ID":"4b171b25-3089-445b-a7a5-65a5c23198be","Type":"ContainerStarted","Data":"f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469"}
Jul 07 17:08:40.857564 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.857211    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:40.857564 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.857240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" event={"ID":"4b171b25-3089-445b-a7a5-65a5c23198be","Type":"ContainerStarted","Data":"e26731a9e8686aaf6281ccbd537e182e2782c6b7c7e32cc58cc55d1345d2891b"}
Jul 07 17:08:40.859116 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.859083    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 07 17:08:40.859378 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.859356    2559 generic.go:356] "Generic (PLEG): container finished" podID="601ec390-60cf-47aa-817c-c708ace44d65" containerID="22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984" exitCode=2
Jul 07 17:08:40.859460 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.859406    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" event={"ID":"601ec390-60cf-47aa-817c-c708ace44d65","Type":"ContainerDied","Data":"22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984"}
Jul 07 17:08:40.875286 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:40.875242    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podStartSLOduration=0.875229032 podStartE2EDuration="875.229032ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:08:40.873404069 +0000 UTC m=+1259.136940725" watchObservedRunningTime="2026-07-07 17:08:40.875229032 +0000 UTC m=+1259.138765662"
Jul 07 17:08:41.863759 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:41.863722    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 07 17:08:42.716069 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:42.715963    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 07 17:08:43.003477 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.003454    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:08:43.129349 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.129324    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601ec390-60cf-47aa-817c-c708ace44d65-proxy-tls\") pod \"601ec390-60cf-47aa-817c-c708ace44d65\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") "
Jul 07 17:08:43.129497 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.129404    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stfcx\" (UniqueName: \"kubernetes.io/projected/601ec390-60cf-47aa-817c-c708ace44d65-kube-api-access-stfcx\") pod \"601ec390-60cf-47aa-817c-c708ace44d65\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") "
Jul 07 17:08:43.129536 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.129516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601ec390-60cf-47aa-817c-c708ace44d65-kube-rbac-proxy-sar-config\") pod \"601ec390-60cf-47aa-817c-c708ace44d65\" (UID: \"601ec390-60cf-47aa-817c-c708ace44d65\") "
Jul 07 17:08:43.129818 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.129790    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601ec390-60cf-47aa-817c-c708ace44d65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "601ec390-60cf-47aa-817c-c708ace44d65" (UID: "601ec390-60cf-47aa-817c-c708ace44d65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:08:43.131310 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.131286    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601ec390-60cf-47aa-817c-c708ace44d65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "601ec390-60cf-47aa-817c-c708ace44d65" (UID: "601ec390-60cf-47aa-817c-c708ace44d65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:08:43.131310 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.131298    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601ec390-60cf-47aa-817c-c708ace44d65-kube-api-access-stfcx" (OuterVolumeSpecName: "kube-api-access-stfcx") pod "601ec390-60cf-47aa-817c-c708ace44d65" (UID: "601ec390-60cf-47aa-817c-c708ace44d65"). InnerVolumeSpecName "kube-api-access-stfcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:08:43.230633 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.230602    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stfcx\" (UniqueName: \"kubernetes.io/projected/601ec390-60cf-47aa-817c-c708ace44d65-kube-api-access-stfcx\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:08:43.230633 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.230627    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/601ec390-60cf-47aa-817c-c708ace44d65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:08:43.230633 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.230637    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/601ec390-60cf-47aa-817c-c708ace44d65-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:08:43.873868 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.873835    2559 generic.go:356] "Generic (PLEG): container finished" podID="601ec390-60cf-47aa-817c-c708ace44d65" containerID="9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275" exitCode=0
Jul 07 17:08:43.874026 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.873885    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"
Jul 07 17:08:43.874026 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.873903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" event={"ID":"601ec390-60cf-47aa-817c-c708ace44d65","Type":"ContainerDied","Data":"9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275"}
Jul 07 17:08:43.874026 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.873950    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6" event={"ID":"601ec390-60cf-47aa-817c-c708ace44d65","Type":"ContainerDied","Data":"78d306f3dc0c5f7d67fd7902468cc856a01240cf3dee77be355b0862b63fab1f"}
Jul 07 17:08:43.874026 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.873995    2559 scope.go:117] "RemoveContainer" containerID="22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984"
Jul 07 17:08:43.881995 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.881964    2559 scope.go:117] "RemoveContainer" containerID="9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275"
Jul 07 17:08:43.888892 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.888873    2559 scope.go:117] "RemoveContainer" containerID="22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984"
Jul 07 17:08:43.889471 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:08:43.889447    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984\": container with ID starting with 22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984 not found: ID does not exist" containerID="22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984"
Jul 07 17:08:43.889636 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.889479    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984"} err="failed to get container status \"22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984\": rpc error: code = NotFound desc = could not find container \"22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984\": container with ID starting with 22e6bb1bf627d4ee52e7c5bfc2bae8c94b514c864e69694a855b96863df34984 not found: ID does not exist"
Jul 07 17:08:43.889636 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.889501    2559 scope.go:117] "RemoveContainer" containerID="9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275"
Jul 07 17:08:43.889831 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:08:43.889802    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275\": container with ID starting with 9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275 not found: ID does not exist" containerID="9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275"
Jul 07 17:08:43.889886 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.889828    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275"} err="failed to get container status \"9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275\": rpc error: code = NotFound desc = could not find container \"9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275\": container with ID starting with 9a76526c8e7dfbb71fe5e1483b5d9e6e91164710ab5c940b15a373f4e93b9275 not found: ID does not exist"
Jul 07 17:08:43.891430 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.891412    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"]
Jul 07 17:08:43.894872 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:43.894849    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8411f-predictor-55547cb9c5-5bbx6"]
Jul 07 17:08:45.335514 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:45.335482    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="601ec390-60cf-47aa-817c-c708ace44d65" path="/var/lib/kubelet/pods/601ec390-60cf-47aa-817c-c708ace44d65/volumes"
Jul 07 17:08:46.868355 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:46.868329    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:08:46.868936 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:46.868909    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 07 17:08:52.716147 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:52.716110    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:08:56.869866 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:08:56.869824    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 07 17:09:04.385127 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:09:04.385065    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:09:04.982133 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:09:04.982099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:09:06.869393 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:09:06.869350    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 07 17:09:16.869608 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:09:16.869569    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 07 17:09:26.870026 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:09:26.869992    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:09:40.159961 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:09:40.159918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:09:41.253132 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:09:41.253095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:11:51.338526 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:11:51.338486    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:12:24.245618 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:12:24.245578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:14:28.554604 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:14:28.554567    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:17:19.229929 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.229895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:17:19.361745 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.361713    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"]
Jul 07 17:17:19.362101 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.362086    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container"
Jul 07 17:17:19.362158 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.362105    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container"
Jul 07 17:17:19.362158 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.362128    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kube-rbac-proxy"
Jul 07 17:17:19.362158 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.362135    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kube-rbac-proxy"
Jul 07 17:17:19.362305 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.362203    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kserve-container"
Jul 07 17:17:19.362305 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.362212    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="601ec390-60cf-47aa-817c-c708ace44d65" containerName="kube-rbac-proxy"
Jul 07 17:17:19.365611 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.365580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:19.367319 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.367294    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"]
Jul 07 17:17:19.367554 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.367530    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container" containerID="cri-o://558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359" gracePeriod=30
Jul 07 17:17:19.367646 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.367589    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kube-rbac-proxy" containerID="cri-o://7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b" gracePeriod=30
Jul 07 17:17:19.369180 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.369157    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f587d-predictor-serving-cert\""
Jul 07 17:17:19.369394 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.369375    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f587d-kube-rbac-proxy-sar-config\""
Jul 07 17:17:19.375449 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.375427    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"]
Jul 07 17:17:19.422338 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.422313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a4f78b-fe8f-47b4-a95f-560ee160bbea-proxy-tls\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:19.422480 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.422365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:19.422480 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.422391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zr5l\" (UniqueName: \"kubernetes.io/projected/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-api-access-2zr5l\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:19.523910 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.523878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:19.524080 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.523921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zr5l\" (UniqueName: \"kubernetes.io/projected/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-api-access-2zr5l\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:19.524139 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.524098    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a4f78b-fe8f-47b4-a95f-560ee160bbea-proxy-tls\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:19.524305 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:17:19.524281    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f587d-predictor-serving-cert: secret "success-200-isvc-f587d-predictor-serving-cert" not found
Jul 07 17:17:19.524382 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:17:19.524370    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a4f78b-fe8f-47b4-a95f-560ee160bbea-proxy-tls podName:59a4f78b-fe8f-47b4-a95f-560ee160bbea nodeName:}" failed. No retries permitted until 2026-07-07 17:17:20.024351508 +0000 UTC m=+1778.287888119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/59a4f78b-fe8f-47b4-a95f-560ee160bbea-proxy-tls") pod "success-200-isvc-f587d-predictor-85f6444b75-74msz" (UID: "59a4f78b-fe8f-47b4-a95f-560ee160bbea") : secret "success-200-isvc-f587d-predictor-serving-cert" not found
Jul 07 17:17:19.524586 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.524568    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:19.532105 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:19.532079    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zr5l\" (UniqueName: \"kubernetes.io/projected/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-api-access-2zr5l\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:20.029491 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:20.029455    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a4f78b-fe8f-47b4-a95f-560ee160bbea-proxy-tls\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:20.031867 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:20.031844    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a4f78b-fe8f-47b4-a95f-560ee160bbea-proxy-tls\") pod \"success-200-isvc-f587d-predictor-85f6444b75-74msz\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") " pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:20.070170 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:20.070143    2559 generic.go:356] "Generic (PLEG): container finished" podID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerID="7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b" exitCode=2
Jul 07 17:17:20.070308 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:20.070223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" event={"ID":"e7bd633f-19f0-4ca2-8394-c2f81f97526f","Type":"ContainerDied","Data":"7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b"}
Jul 07 17:17:20.287879 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:20.287791    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:20.416609 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:20.416585    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"]
Jul 07 17:17:20.418735 ip-10-0-142-63 kubenswrapper[2559]: W0707 17:17:20.418694    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a4f78b_fe8f_47b4_a95f_560ee160bbea.slice/crio-7f6495ff8017b6e257218c3493f49cb4f1592f561a1f1039751e99d59511b803 WatchSource:0}: Error finding container 7f6495ff8017b6e257218c3493f49cb4f1592f561a1f1039751e99d59511b803: Status 404 returned error can't find the container with id 7f6495ff8017b6e257218c3493f49cb4f1592f561a1f1039751e99d59511b803
Jul 07 17:17:20.837722 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:20.837688    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:17:21.076948 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:21.076916    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" event={"ID":"59a4f78b-fe8f-47b4-a95f-560ee160bbea","Type":"ContainerStarted","Data":"29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a"}
Jul 07 17:17:21.077137 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:21.076995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" event={"ID":"59a4f78b-fe8f-47b4-a95f-560ee160bbea","Type":"ContainerStarted","Data":"7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a"}
Jul 07 17:17:21.077137 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:21.077119    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:21.077257 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:21.077172    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:21.077257 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:21.077204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" event={"ID":"59a4f78b-fe8f-47b4-a95f-560ee160bbea","Type":"ContainerStarted","Data":"7f6495ff8017b6e257218c3493f49cb4f1592f561a1f1039751e99d59511b803"}
Jul 07 17:17:21.078375 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:21.078355    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 07 17:17:21.099722 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:21.099640    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podStartSLOduration=2.099627159 podStartE2EDuration="2.099627159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:17:21.098041436 +0000 UTC m=+1779.361578071" watchObservedRunningTime="2026-07-07 17:17:21.099627159 +0000 UTC m=+1779.363163789"
Jul 07 17:17:22.082374 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.082336    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 07 17:17:22.421987 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.421948    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:17:22.449496 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.449467    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7bd633f-19f0-4ca2-8394-c2f81f97526f-proxy-tls\") pod \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") "
Jul 07 17:17:22.449616 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.449523    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5jlv\" (UniqueName: \"kubernetes.io/projected/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-api-access-j5jlv\") pod \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") "
Jul 07 17:17:22.449616 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.449552    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-rbac-proxy-sar-config\") pod \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\" (UID: \"e7bd633f-19f0-4ca2-8394-c2f81f97526f\") "
Jul 07 17:17:22.449947 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.449921    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7bd633f-19f0-4ca2-8394-c2f81f97526f" (UID: "e7bd633f-19f0-4ca2-8394-c2f81f97526f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:17:22.451577 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.451550    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7bd633f-19f0-4ca2-8394-c2f81f97526f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7bd633f-19f0-4ca2-8394-c2f81f97526f" (UID: "e7bd633f-19f0-4ca2-8394-c2f81f97526f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:17:22.451577 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.451550    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-api-access-j5jlv" (OuterVolumeSpecName: "kube-api-access-j5jlv") pod "e7bd633f-19f0-4ca2-8394-c2f81f97526f" (UID: "e7bd633f-19f0-4ca2-8394-c2f81f97526f"). InnerVolumeSpecName "kube-api-access-j5jlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:17:22.550923 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.550897    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7bd633f-19f0-4ca2-8394-c2f81f97526f-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:17:22.550923 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.550920    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jlv\" (UniqueName: \"kubernetes.io/projected/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-api-access-j5jlv\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:17:22.551075 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:22.550930    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7bd633f-19f0-4ca2-8394-c2f81f97526f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:17:23.087424 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.087387    2559 generic.go:356] "Generic (PLEG): container finished" podID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerID="558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359" exitCode=0
Jul 07 17:17:23.087795 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.087438    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"
Jul 07 17:17:23.087795 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.087448    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" event={"ID":"e7bd633f-19f0-4ca2-8394-c2f81f97526f","Type":"ContainerDied","Data":"558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359"}
Jul 07 17:17:23.087795 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.087494    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq" event={"ID":"e7bd633f-19f0-4ca2-8394-c2f81f97526f","Type":"ContainerDied","Data":"53a4620fa5f9130c1e3c9d1a92083f051c111616c084177f84cd940471d2bd41"}
Jul 07 17:17:23.087795 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.087521    2559 scope.go:117] "RemoveContainer" containerID="7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b"
Jul 07 17:17:23.095773 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.095755    2559 scope.go:117] "RemoveContainer" containerID="558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359"
Jul 07 17:17:23.102823 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.102805    2559 scope.go:117] "RemoveContainer" containerID="7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b"
Jul 07 17:17:23.103090 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:17:23.103070    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b\": container with ID starting with 7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b not found: ID does not exist" containerID="7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b"
Jul 07 17:17:23.103146 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.103097    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b"} err="failed to get container status \"7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b\": rpc error: code = NotFound desc = could not find container \"7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b\": container with ID starting with 7bee38fb3b917de16f39a04f59e7aa0daa55e3095ced3a451a16e050be32ea4b not found: ID does not exist"
Jul 07 17:17:23.103146 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.103113    2559 scope.go:117] "RemoveContainer" containerID="558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359"
Jul 07 17:17:23.103346 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:17:23.103328    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359\": container with ID starting with 558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359 not found: ID does not exist" containerID="558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359"
Jul 07 17:17:23.103388 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.103352    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359"} err="failed to get container status \"558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359\": rpc error: code = NotFound desc = could not find container \"558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359\": container with ID starting with 558b8453e1b38f8495f3cf5b75d6ad0d19d3591ce196b0d27499d9aac0513359 not found: ID does not exist"
Jul 07 17:17:23.107242 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.107221    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"]
Jul 07 17:17:23.111236 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.111217    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f8ada-predictor-b77b54b8c-nkpmq"]
Jul 07 17:17:23.336339 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:23.336305    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" path="/var/lib/kubelet/pods/e7bd633f-19f0-4ca2-8394-c2f81f97526f/volumes"
Jul 07 17:17:27.086787 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:27.086758    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:17:27.087257 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:27.087229    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 07 17:17:37.087351 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:37.087309    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 07 17:17:47.087479 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:47.087433    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 07 17:17:54.834264 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.834220    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:17:54.935640 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.935608    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"]
Jul 07 17:17:54.935906 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.935866    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container" containerID="cri-o://f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469" gracePeriod=30
Jul 07 17:17:54.936026 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.935954    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kube-rbac-proxy" containerID="cri-o://fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843" gracePeriod=30
Jul 07 17:17:54.947962 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.947784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:17:54.952128 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.952107    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"]
Jul 07 17:17:54.952434 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.952423    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kube-rbac-proxy"
Jul 07 17:17:54.952473 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.952436    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kube-rbac-proxy"
Jul 07 17:17:54.952473 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.952450    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container"
Jul 07 17:17:54.952473 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.952456    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container"
Jul 07 17:17:54.952569 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.952515    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kube-rbac-proxy"
Jul 07 17:17:54.952569 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.952527    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7bd633f-19f0-4ca2-8394-c2f81f97526f" containerName="kserve-container"
Jul 07 17:17:54.956459 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.956440    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:54.958802 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.958777    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1a68d-kube-rbac-proxy-sar-config\""
Jul 07 17:17:54.958900 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.958800    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1a68d-predictor-serving-cert\""
Jul 07 17:17:54.965839 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:54.965818    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"]
Jul 07 17:17:55.044847 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.044804    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:17:55.108047 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.107939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d79b682-bdc2-4c76-9a46-18087eda25d0-proxy-tls\") pod \"success-200-isvc-1a68d-predictor-784765f895-hgr6w\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") " pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.108047 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.107998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1a68d-predictor-784765f895-hgr6w\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") " pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.108229 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.108129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v8pp\" (UniqueName: \"kubernetes.io/projected/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-api-access-8v8pp\") pod \"success-200-isvc-1a68d-predictor-784765f895-hgr6w\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") " pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.116297 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.116265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:17:55.208856 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.208823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d79b682-bdc2-4c76-9a46-18087eda25d0-proxy-tls\") pod \"success-200-isvc-1a68d-predictor-784765f895-hgr6w\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") " pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.208856 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.208856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1a68d-predictor-784765f895-hgr6w\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") " pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.209126 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.208914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v8pp\" (UniqueName: \"kubernetes.io/projected/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-api-access-8v8pp\") pod \"success-200-isvc-1a68d-predictor-784765f895-hgr6w\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") " pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.209742 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.209696    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1a68d-predictor-784765f895-hgr6w\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") " pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.210064 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.210039    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:17:55.211823 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.211792    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d79b682-bdc2-4c76-9a46-18087eda25d0-proxy-tls\") pod \"success-200-isvc-1a68d-predictor-784765f895-hgr6w\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") " pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.216645 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.216616    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v8pp\" (UniqueName: \"kubernetes.io/projected/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-api-access-8v8pp\") pod \"success-200-isvc-1a68d-predictor-784765f895-hgr6w\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") " pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.228363 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.228340    2559 generic.go:356] "Generic (PLEG): container finished" podID="4b171b25-3089-445b-a7a5-65a5c23198be" containerID="fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843" exitCode=2
Jul 07 17:17:55.228450 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.228409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" event={"ID":"4b171b25-3089-445b-a7a5-65a5c23198be","Type":"ContainerDied","Data":"fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843"}
Jul 07 17:17:55.267685 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.267661    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:55.399859 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.399834    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"]
Jul 07 17:17:55.401915 ip-10-0-142-63 kubenswrapper[2559]: W0707 17:17:55.401887    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d79b682_bdc2_4c76_9a46_18087eda25d0.slice/crio-4422fbd718abdcf7e591489f86b77f71bb391419d311c143f8ea809238a724e6 WatchSource:0}: Error finding container 4422fbd718abdcf7e591489f86b77f71bb391419d311c143f8ea809238a724e6: Status 404 returned error can't find the container with id 4422fbd718abdcf7e591489f86b77f71bb391419d311c143f8ea809238a724e6
Jul 07 17:17:55.506695 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:55.506652    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:17:56.234286 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:56.234248    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" event={"ID":"9d79b682-bdc2-4c76-9a46-18087eda25d0","Type":"ContainerStarted","Data":"e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61"}
Jul 07 17:17:56.234674 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:56.234313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" event={"ID":"9d79b682-bdc2-4c76-9a46-18087eda25d0","Type":"ContainerStarted","Data":"6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d"}
Jul 07 17:17:56.234674 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:56.234421    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:56.234674 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:56.234460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" event={"ID":"9d79b682-bdc2-4c76-9a46-18087eda25d0","Type":"ContainerStarted","Data":"4422fbd718abdcf7e591489f86b77f71bb391419d311c143f8ea809238a724e6"}
Jul 07 17:17:56.234674 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:56.234592    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:17:56.236046 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:56.236017    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 07 17:17:56.251131 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:56.251090    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podStartSLOduration=2.251077966 podStartE2EDuration="2.251077966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:17:56.24964497 +0000 UTC m=+1814.513181621" watchObservedRunningTime="2026-07-07 17:17:56.251077966 +0000 UTC m=+1814.514614597"
Jul 07 17:17:56.864851 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:56.864808    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 07 17:17:56.869183 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:56.869155    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 07 17:17:57.087724 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:57.087682    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 07 17:17:57.238907 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:57.238815    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 07 17:17:57.890569 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:57.890546    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:17:58.030985 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.030956    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw7sl\" (UniqueName: \"kubernetes.io/projected/4b171b25-3089-445b-a7a5-65a5c23198be-kube-api-access-pw7sl\") pod \"4b171b25-3089-445b-a7a5-65a5c23198be\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") "
Jul 07 17:17:58.031126 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.031049    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b171b25-3089-445b-a7a5-65a5c23198be-kube-rbac-proxy-sar-config\") pod \"4b171b25-3089-445b-a7a5-65a5c23198be\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") "
Jul 07 17:17:58.031126 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.031076    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b171b25-3089-445b-a7a5-65a5c23198be-proxy-tls\") pod \"4b171b25-3089-445b-a7a5-65a5c23198be\" (UID: \"4b171b25-3089-445b-a7a5-65a5c23198be\") "
Jul 07 17:17:58.031341 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.031319    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b171b25-3089-445b-a7a5-65a5c23198be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b171b25-3089-445b-a7a5-65a5c23198be" (UID: "4b171b25-3089-445b-a7a5-65a5c23198be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:17:58.033057 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.033029    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b171b25-3089-445b-a7a5-65a5c23198be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b171b25-3089-445b-a7a5-65a5c23198be" (UID: "4b171b25-3089-445b-a7a5-65a5c23198be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:17:58.033182 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.033161    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b171b25-3089-445b-a7a5-65a5c23198be-kube-api-access-pw7sl" (OuterVolumeSpecName: "kube-api-access-pw7sl") pod "4b171b25-3089-445b-a7a5-65a5c23198be" (UID: "4b171b25-3089-445b-a7a5-65a5c23198be"). InnerVolumeSpecName "kube-api-access-pw7sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:17:58.132025 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.131970    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw7sl\" (UniqueName: \"kubernetes.io/projected/4b171b25-3089-445b-a7a5-65a5c23198be-kube-api-access-pw7sl\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:17:58.132025 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.132022    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b171b25-3089-445b-a7a5-65a5c23198be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:17:58.132156 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.132031    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b171b25-3089-445b-a7a5-65a5c23198be-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:17:58.244816 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.244777    2559 generic.go:356] "Generic (PLEG): container finished" podID="4b171b25-3089-445b-a7a5-65a5c23198be" containerID="f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469" exitCode=0
Jul 07 17:17:58.245232 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.244833    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"
Jul 07 17:17:58.245232 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.244856    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" event={"ID":"4b171b25-3089-445b-a7a5-65a5c23198be","Type":"ContainerDied","Data":"f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469"}
Jul 07 17:17:58.245232 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.244916    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj" event={"ID":"4b171b25-3089-445b-a7a5-65a5c23198be","Type":"ContainerDied","Data":"e26731a9e8686aaf6281ccbd537e182e2782c6b7c7e32cc58cc55d1345d2891b"}
Jul 07 17:17:58.245232 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.244942    2559 scope.go:117] "RemoveContainer" containerID="fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843"
Jul 07 17:17:58.245484 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.245456    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 07 17:17:58.253871 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.253852    2559 scope.go:117] "RemoveContainer" containerID="f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469"
Jul 07 17:17:58.261467 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.261449    2559 scope.go:117] "RemoveContainer" containerID="fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843"
Jul 07 17:17:58.261748 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:17:58.261726    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843\": container with ID starting with fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843 not found: ID does not exist" containerID="fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843"
Jul 07 17:17:58.261838 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.261757    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843"} err="failed to get container status \"fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843\": rpc error: code = NotFound desc = could not find container \"fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843\": container with ID starting with fb6759a8308a857b3484ee58f251b25df1db0fc3f3580d0f46e2f80da0a2d843 not found: ID does not exist"
Jul 07 17:17:58.261838 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.261773    2559 scope.go:117] "RemoveContainer" containerID="f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469"
Jul 07 17:17:58.262059 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:17:58.262032    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469\": container with ID starting with f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469 not found: ID does not exist" containerID="f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469"
Jul 07 17:17:58.262138 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.262068    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469"} err="failed to get container status \"f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469\": rpc error: code = NotFound desc = could not find container \"f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469\": container with ID starting with f8ca5302809ca80f886ee625407be5cd472d66d11787b3cf57ee299f0e0e3469 not found: ID does not exist"
Jul 07 17:17:58.266830 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.266807    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"]
Jul 07 17:17:58.272857 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:58.272838    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3653b-predictor-849b9cdd5-4p4pj"]
Jul 07 17:17:59.336332 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:17:59.336302    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" path="/var/lib/kubelet/pods/4b171b25-3089-445b-a7a5-65a5c23198be/volumes"
Jul 07 17:18:03.250238 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:03.250209    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:18:03.250671 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:03.250644    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 07 17:18:07.087873 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:07.087846    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:18:13.251212 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:13.251172    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 07 17:18:19.413861 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:19.413827    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:19.558367 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:19.558329    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:23.250605 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:23.250559    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 07 17:18:29.509557 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.509511    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:29.600434 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.600399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:29.626730 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.626693    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"]
Jul 07 17:18:29.627010 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.626962    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container" containerID="cri-o://7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a" gracePeriod=30
Jul 07 17:18:29.627010 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.626969    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kube-rbac-proxy" containerID="cri-o://29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a" gracePeriod=30
Jul 07 17:18:29.647812 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.647786    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"]
Jul 07 17:18:29.648167 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.648153    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container"
Jul 07 17:18:29.648219 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.648169    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container"
Jul 07 17:18:29.648219 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.648181    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kube-rbac-proxy"
Jul 07 17:18:29.648219 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.648187    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kube-rbac-proxy"
Jul 07 17:18:29.648311 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.648241    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kube-rbac-proxy"
Jul 07 17:18:29.648311 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.648250    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b171b25-3089-445b-a7a5-65a5c23198be" containerName="kserve-container"
Jul 07 17:18:29.651521 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.651500    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:29.653890 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.653874    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f025d-kube-rbac-proxy-sar-config\""
Jul 07 17:18:29.653963 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.653948    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f025d-predictor-serving-cert\""
Jul 07 17:18:29.662686 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.662665    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"]
Jul 07 17:18:29.672898 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.672869    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:29.673026 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.672913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-proxy-tls\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:29.673026 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.673019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqzg\" (UniqueName: \"kubernetes.io/projected/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-api-access-vqqzg\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:29.713583 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.713555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:29.773873 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.773811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqzg\" (UniqueName: \"kubernetes.io/projected/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-api-access-vqqzg\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:29.774015 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.773893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:29.774015 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.773931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-proxy-tls\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:29.774130 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:18:29.774081    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f025d-predictor-serving-cert: secret "success-200-isvc-f025d-predictor-serving-cert" not found
Jul 07 17:18:29.774177 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:18:29.774161    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-proxy-tls podName:44e8ca15-e9b2-4ee0-bc59-46cde78e5796 nodeName:}" failed. No retries permitted until 2026-07-07 17:18:30.274139532 +0000 UTC m=+1848.537676149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-proxy-tls") pod "success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" (UID: "44e8ca15-e9b2-4ee0-bc59-46cde78e5796") : secret "success-200-isvc-f025d-predictor-serving-cert" not found
Jul 07 17:18:29.774581 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.774563    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:29.784438 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.784417    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqzg\" (UniqueName: \"kubernetes.io/projected/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-api-access-vqqzg\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:29.811917 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.811893    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:29.912501 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:29.912459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:30.101764 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:30.101680    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:30.278189 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:30.278158    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-proxy-tls\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:30.280593 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:30.280562    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-proxy-tls\") pod \"success-200-isvc-f025d-predictor-5bc769fb45-7c9v6\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") " pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:30.383062 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:30.382966    2559 generic.go:356] "Generic (PLEG): container finished" podID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerID="29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a" exitCode=2
Jul 07 17:18:30.383062 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:30.383047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" event={"ID":"59a4f78b-fe8f-47b4-a95f-560ee160bbea","Type":"ContainerDied","Data":"29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a"}
Jul 07 17:18:30.562192 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:30.562158    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:30.690553 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:30.690498    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"]
Jul 07 17:18:30.692739 ip-10-0-142-63 kubenswrapper[2559]: W0707 17:18:30.692712    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e8ca15_e9b2_4ee0_bc59_46cde78e5796.slice/crio-31f1f3c01c61d4c463d7817869aeca77caf5a4b9948216fe5983a8d062d973d0 WatchSource:0}: Error finding container 31f1f3c01c61d4c463d7817869aeca77caf5a4b9948216fe5983a8d062d973d0: Status 404 returned error can't find the container with id 31f1f3c01c61d4c463d7817869aeca77caf5a4b9948216fe5983a8d062d973d0
Jul 07 17:18:31.388478 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:31.388444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" event={"ID":"44e8ca15-e9b2-4ee0-bc59-46cde78e5796","Type":"ContainerStarted","Data":"bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd"}
Jul 07 17:18:31.388653 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:31.388507    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:31.388653 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:31.388540    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:31.388653 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:31.388565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" event={"ID":"44e8ca15-e9b2-4ee0-bc59-46cde78e5796","Type":"ContainerStarted","Data":"03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89"}
Jul 07 17:18:31.388653 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:31.388592    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" event={"ID":"44e8ca15-e9b2-4ee0-bc59-46cde78e5796","Type":"ContainerStarted","Data":"31f1f3c01c61d4c463d7817869aeca77caf5a4b9948216fe5983a8d062d973d0"}
Jul 07 17:18:31.389845 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:31.389817    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 07 17:18:31.406659 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:31.406620    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" podStartSLOduration=2.406607392 podStartE2EDuration="2.406607392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:18:31.405453063 +0000 UTC m=+1849.668989720" watchObservedRunningTime="2026-07-07 17:18:31.406607392 +0000 UTC m=+1849.670144024"
Jul 07 17:18:32.082902 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.082863    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 07 17:18:32.394039 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.393941    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 07 17:18:32.781031 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.781008    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:18:32.796810 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.796786    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zr5l\" (UniqueName: \"kubernetes.io/projected/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-api-access-2zr5l\") pod \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") "
Jul 07 17:18:32.796907 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.796822    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-rbac-proxy-sar-config\") pod \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") "
Jul 07 17:18:32.796907 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.796867    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a4f78b-fe8f-47b4-a95f-560ee160bbea-proxy-tls\") pod \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\" (UID: \"59a4f78b-fe8f-47b4-a95f-560ee160bbea\") "
Jul 07 17:18:32.797260 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.797233    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59a4f78b-fe8f-47b4-a95f-560ee160bbea" (UID: "59a4f78b-fe8f-47b4-a95f-560ee160bbea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:18:32.799201 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.799178    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-api-access-2zr5l" (OuterVolumeSpecName: "kube-api-access-2zr5l") pod "59a4f78b-fe8f-47b4-a95f-560ee160bbea" (UID: "59a4f78b-fe8f-47b4-a95f-560ee160bbea"). InnerVolumeSpecName "kube-api-access-2zr5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:18:32.799404 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.799375    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a4f78b-fe8f-47b4-a95f-560ee160bbea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59a4f78b-fe8f-47b4-a95f-560ee160bbea" (UID: "59a4f78b-fe8f-47b4-a95f-560ee160bbea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:18:32.897758 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.897688    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a4f78b-fe8f-47b4-a95f-560ee160bbea-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:18:32.897758 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.897717    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zr5l\" (UniqueName: \"kubernetes.io/projected/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-api-access-2zr5l\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:18:32.897758 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:32.897729    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a4f78b-fe8f-47b4-a95f-560ee160bbea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:18:33.251167 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.251083    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 07 17:18:33.399298 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.399266    2559 generic.go:356] "Generic (PLEG): container finished" podID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerID="7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a" exitCode=0
Jul 07 17:18:33.399428 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.399313    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"
Jul 07 17:18:33.399428 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.399324    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" event={"ID":"59a4f78b-fe8f-47b4-a95f-560ee160bbea","Type":"ContainerDied","Data":"7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a"}
Jul 07 17:18:33.399428 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.399368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz" event={"ID":"59a4f78b-fe8f-47b4-a95f-560ee160bbea","Type":"ContainerDied","Data":"7f6495ff8017b6e257218c3493f49cb4f1592f561a1f1039751e99d59511b803"}
Jul 07 17:18:33.399428 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.399399    2559 scope.go:117] "RemoveContainer" containerID="29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a"
Jul 07 17:18:33.429898 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.429869    2559 scope.go:117] "RemoveContainer" containerID="7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a"
Jul 07 17:18:33.437678 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.437656    2559 scope.go:117] "RemoveContainer" containerID="29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a"
Jul 07 17:18:33.437950 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:18:33.437925    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a\": container with ID starting with 29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a not found: ID does not exist" containerID="29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a"
Jul 07 17:18:33.438038 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.437962    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a"} err="failed to get container status \"29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a\": rpc error: code = NotFound desc = could not find container \"29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a\": container with ID starting with 29a5464434bffe53518f785b0e744ad5252b7b4a200dd8863f4fa337cfe8554a not found: ID does not exist"
Jul 07 17:18:33.438038 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.437997    2559 scope.go:117] "RemoveContainer" containerID="7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a"
Jul 07 17:18:33.438263 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:18:33.438248    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a\": container with ID starting with 7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a not found: ID does not exist" containerID="7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a"
Jul 07 17:18:33.438306 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.438269    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a"} err="failed to get container status \"7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a\": rpc error: code = NotFound desc = could not find container \"7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a\": container with ID starting with 7bd13a219fb292d955c2e5cb4d1f04486ede4b11502aabfb7cbb2796d55eeb5a not found: ID does not exist"
Jul 07 17:18:33.439129 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.439106    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"]
Jul 07 17:18:33.446384 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:33.446359    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f587d-predictor-85f6444b75-74msz"]
Jul 07 17:18:35.336740 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:35.336704    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" path="/var/lib/kubelet/pods/59a4f78b-fe8f-47b4-a95f-560ee160bbea/volumes"
Jul 07 17:18:37.397858 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:37.397828    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:18:37.398300 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:37.398276    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 07 17:18:43.251624 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:43.251593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:18:47.398691 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:47.398655    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 07 17:18:55.063383 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:55.063345    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:55.388549 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:55.388450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:18:57.399094 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:18:57.399052    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 07 17:19:07.398316 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:19:07.398228    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 07 17:19:17.398853 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:19:17.398821    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:19:29.733127 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:19:29.733091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:19:29.832560 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:19:29.832525    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:19:31.896868 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:19:31.896835    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:19:31.946331 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:19:31.946301    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:22:19.222813 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:22:19.222738    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:26:08.456157 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:26:08.456098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:27:36.149761 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:36.149670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:27:44.324167 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:44.324126    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:27:44.416819 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:44.416787    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"]
Jul 07 17:27:44.417095 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:44.417058    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container" containerID="cri-o://03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89" gracePeriod=30
Jul 07 17:27:44.417233 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:44.417102    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kube-rbac-proxy" containerID="cri-o://bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd" gracePeriod=30
Jul 07 17:27:44.443990 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:44.443925    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:27:44.512531 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:44.512509    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:27:44.723112 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:44.723016    2559 generic.go:356] "Generic (PLEG): container finished" podID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerID="bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd" exitCode=2
Jul 07 17:27:44.723112 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:44.723099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" event={"ID":"44e8ca15-e9b2-4ee0-bc59-46cde78e5796","Type":"ContainerDied","Data":"bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd"}
Jul 07 17:27:47.164441 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.164418    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:27:47.250722 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.250695    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-proxy-tls\") pod \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") "
Jul 07 17:27:47.250857 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.250739    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqqzg\" (UniqueName: \"kubernetes.io/projected/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-api-access-vqqzg\") pod \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") "
Jul 07 17:27:47.250857 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.250790    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-rbac-proxy-sar-config\") pod \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\" (UID: \"44e8ca15-e9b2-4ee0-bc59-46cde78e5796\") "
Jul 07 17:27:47.251285 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.251161    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44e8ca15-e9b2-4ee0-bc59-46cde78e5796" (UID: "44e8ca15-e9b2-4ee0-bc59-46cde78e5796"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:27:47.252713 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.252692    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44e8ca15-e9b2-4ee0-bc59-46cde78e5796" (UID: "44e8ca15-e9b2-4ee0-bc59-46cde78e5796"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:27:47.252763 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.252745    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-api-access-vqqzg" (OuterVolumeSpecName: "kube-api-access-vqqzg") pod "44e8ca15-e9b2-4ee0-bc59-46cde78e5796" (UID: "44e8ca15-e9b2-4ee0-bc59-46cde78e5796"). InnerVolumeSpecName "kube-api-access-vqqzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:27:47.352029 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.352005    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:27:47.352029 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.352026    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqzg\" (UniqueName: \"kubernetes.io/projected/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-api-access-vqqzg\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:27:47.352153 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.352037    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44e8ca15-e9b2-4ee0-bc59-46cde78e5796-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:27:47.739576 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.739497    2559 generic.go:356] "Generic (PLEG): container finished" podID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerID="03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89" exitCode=0
Jul 07 17:27:47.739576 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.739547    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"
Jul 07 17:27:47.739765 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.739548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" event={"ID":"44e8ca15-e9b2-4ee0-bc59-46cde78e5796","Type":"ContainerDied","Data":"03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89"}
Jul 07 17:27:47.739765 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.739679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6" event={"ID":"44e8ca15-e9b2-4ee0-bc59-46cde78e5796","Type":"ContainerDied","Data":"31f1f3c01c61d4c463d7817869aeca77caf5a4b9948216fe5983a8d062d973d0"}
Jul 07 17:27:47.739765 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.739710    2559 scope.go:117] "RemoveContainer" containerID="bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd"
Jul 07 17:27:47.753394 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.753366    2559 scope.go:117] "RemoveContainer" containerID="03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89"
Jul 07 17:27:47.759050 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.759029    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"]
Jul 07 17:27:47.761579 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.761564    2559 scope.go:117] "RemoveContainer" containerID="bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd"
Jul 07 17:27:47.761847 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:27:47.761827    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd\": container with ID starting with bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd not found: ID does not exist" containerID="bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd"
Jul 07 17:27:47.761918 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.761856    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd"} err="failed to get container status \"bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd\": rpc error: code = NotFound desc = could not find container \"bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd\": container with ID starting with bae189dafc58df184317436bd9552e324772628a22826284dfc87f5b41447ecd not found: ID does not exist"
Jul 07 17:27:47.761918 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.761903    2559 scope.go:117] "RemoveContainer" containerID="03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89"
Jul 07 17:27:47.762239 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:27:47.762215    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89\": container with ID starting with 03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89 not found: ID does not exist" containerID="03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89"
Jul 07 17:27:47.762313 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.762247    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89"} err="failed to get container status \"03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89\": rpc error: code = NotFound desc = could not find container \"03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89\": container with ID starting with 03b532c48f5a05ddc4c80c39cfe4ecc0ff092e4f19dfa79b79de9e91d8c59d89 not found: ID does not exist"
Jul 07 17:27:47.762516 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:47.762499    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f025d-predictor-5bc769fb45-7c9v6"]
Jul 07 17:27:49.336165 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:27:49.336135    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" path="/var/lib/kubelet/pods/44e8ca15-e9b2-4ee0-bc59-46cde78e5796/volumes"
Jul 07 17:29:07.705025 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:29:07.704969    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:29:18.948810 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:29:18.948778    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:31:28.559765 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:31:28.559727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:11.039776 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:11.039729    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:14.489402 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:14.489365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:14.614705 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:14.614668    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"]
Jul 07 17:35:14.614953 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:14.614923    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container" containerID="cri-o://6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d" gracePeriod=30
Jul 07 17:35:14.615113 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:14.615009    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kube-rbac-proxy" containerID="cri-o://e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61" gracePeriod=30
Jul 07 17:35:14.634847 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:14.634824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:14.697895 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:14.697869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:15.455842 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:15.455814    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:15.519093 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:15.519063    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:15.629881 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:15.629849    2559 generic.go:356] "Generic (PLEG): container finished" podID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerID="e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61" exitCode=2
Jul 07 17:35:15.630047 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:15.629913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" event={"ID":"9d79b682-bdc2-4c76-9a46-18087eda25d0","Type":"ContainerDied","Data":"e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61"}
Jul 07 17:35:17.258910 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.258886    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:35:17.363295 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.363267    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-rbac-proxy-sar-config\") pod \"9d79b682-bdc2-4c76-9a46-18087eda25d0\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") "
Jul 07 17:35:17.363477 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.363300    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d79b682-bdc2-4c76-9a46-18087eda25d0-proxy-tls\") pod \"9d79b682-bdc2-4c76-9a46-18087eda25d0\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") "
Jul 07 17:35:17.363477 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.363318    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v8pp\" (UniqueName: \"kubernetes.io/projected/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-api-access-8v8pp\") pod \"9d79b682-bdc2-4c76-9a46-18087eda25d0\" (UID: \"9d79b682-bdc2-4c76-9a46-18087eda25d0\") "
Jul 07 17:35:17.363618 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.363587    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d79b682-bdc2-4c76-9a46-18087eda25d0" (UID: "9d79b682-bdc2-4c76-9a46-18087eda25d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:35:17.365313 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.365279    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-api-access-8v8pp" (OuterVolumeSpecName: "kube-api-access-8v8pp") pod "9d79b682-bdc2-4c76-9a46-18087eda25d0" (UID: "9d79b682-bdc2-4c76-9a46-18087eda25d0"). InnerVolumeSpecName "kube-api-access-8v8pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:35:17.365426 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.365308    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d79b682-bdc2-4c76-9a46-18087eda25d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d79b682-bdc2-4c76-9a46-18087eda25d0" (UID: "9d79b682-bdc2-4c76-9a46-18087eda25d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:35:17.463740 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.463706    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:35:17.463740 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.463740    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d79b682-bdc2-4c76-9a46-18087eda25d0-proxy-tls\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:35:17.463883 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.463777    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v8pp\" (UniqueName: \"kubernetes.io/projected/9d79b682-bdc2-4c76-9a46-18087eda25d0-kube-api-access-8v8pp\") on node \"ip-10-0-142-63.ec2.internal\" DevicePath \"\""
Jul 07 17:35:17.639612 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.639548    2559 generic.go:356] "Generic (PLEG): container finished" podID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerID="6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d" exitCode=0
Jul 07 17:35:17.639612 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.639589    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"
Jul 07 17:35:17.639740 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.639630    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" event={"ID":"9d79b682-bdc2-4c76-9a46-18087eda25d0","Type":"ContainerDied","Data":"6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d"}
Jul 07 17:35:17.639740 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.639689    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w" event={"ID":"9d79b682-bdc2-4c76-9a46-18087eda25d0","Type":"ContainerDied","Data":"4422fbd718abdcf7e591489f86b77f71bb391419d311c143f8ea809238a724e6"}
Jul 07 17:35:17.639740 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.639715    2559 scope.go:117] "RemoveContainer" containerID="e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61"
Jul 07 17:35:17.648467 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.648449    2559 scope.go:117] "RemoveContainer" containerID="6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d"
Jul 07 17:35:17.655507 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.655491    2559 scope.go:117] "RemoveContainer" containerID="e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61"
Jul 07 17:35:17.655733 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:35:17.655713    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61\": container with ID starting with e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61 not found: ID does not exist" containerID="e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61"
Jul 07 17:35:17.655792 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.655740    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61"} err="failed to get container status \"e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61\": rpc error: code = NotFound desc = could not find container \"e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61\": container with ID starting with e5a04b5876bfeed9b1755d70f681df74c41dd281497975dafe1afcfb7c835a61 not found: ID does not exist"
Jul 07 17:35:17.655792 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.655756    2559 scope.go:117] "RemoveContainer" containerID="6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d"
Jul 07 17:35:17.656009 ip-10-0-142-63 kubenswrapper[2559]: E0707 17:35:17.655965    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d\": container with ID starting with 6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d not found: ID does not exist" containerID="6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d"
Jul 07 17:35:17.656105 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.656013    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d"} err="failed to get container status \"6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d\": rpc error: code = NotFound desc = could not find container \"6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d\": container with ID starting with 6da127e162abc37c432943d5763fc668489fc68a331e3a562f56367eb76fd19d not found: ID does not exist"
Jul 07 17:35:17.659810 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.659783    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"]
Jul 07 17:35:17.661771 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:17.661751    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1a68d-predictor-784765f895-hgr6w"]
Jul 07 17:35:19.335988 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:19.335942    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" path="/var/lib/kubelet/pods/9d79b682-bdc2-4c76-9a46-18087eda25d0/volumes"
Jul 07 17:35:45.963203 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:45.963165    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:46.186522 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:46.186492    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:46.286685 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:46.286630    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:48.908926 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:48.908892    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c57bv_d273870c-4ed6-44f4-be7c-669c1c013396/global-pull-secret-syncer/0.log"
Jul 07 17:35:49.031109 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:49.031075    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log"
Jul 07 17:35:49.167236 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:49.167156    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-63.ec2.internal_512194b8ab62074e01ca53d7650baf23/haproxy/0.log"
Jul 07 17:35:51.650130 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:51.650098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:51.711779 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:51.711753    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mtp24_374eb757-0763-44cb-864e-3145f83715a4/konnectivity-agent/0.log" limit=1024
Jul 07 17:35:53.943074 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:53.943043    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-gfdb6_eb0a2918-c0c2-4900-a335-ccb31870a05b/kube-state-metrics/0.log"
Jul 07 17:35:53.973887 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:53.973864    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-gfdb6_eb0a2918-c0c2-4900-a335-ccb31870a05b/kube-rbac-proxy-main/0.log"
Jul 07 17:35:53.994740 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:53.994718    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-gfdb6_eb0a2918-c0c2-4900-a335-ccb31870a05b/kube-rbac-proxy-self/0.log"
Jul 07 17:35:54.050795 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.050775    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-6nxx2_332c6a89-f7cb-4c02-8e2b-a3d3c17f5a4e/monitoring-plugin/0.log"
Jul 07 17:35:54.154166 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.154136    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8s562_afc81d20-2e49-4284-96ed-866c03196411/node-exporter/0.log"
Jul 07 17:35:54.173930 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.173908    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8s562_afc81d20-2e49-4284-96ed-866c03196411/kube-rbac-proxy/0.log"
Jul 07 17:35:54.195867 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.195797    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8s562_afc81d20-2e49-4284-96ed-866c03196411/init-textfile/0.log"
Jul 07 17:35:54.297950 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.297924    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mkn4w_81b2546a-97bc-4af2-b109-5f05f40f8e23/kube-rbac-proxy-main/0.log"
Jul 07 17:35:54.319083 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.319056    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mkn4w_81b2546a-97bc-4af2-b109-5f05f40f8e23/kube-rbac-proxy-self/0.log"
Jul 07 17:35:54.339639 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.339622    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-mkn4w_81b2546a-97bc-4af2-b109-5f05f40f8e23/openshift-state-metrics/0.log"
Jul 07 17:35:54.735766 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.735733    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79c9458997-rrnwn_17f6c6b7-c8eb-4101-bdcc-bde59e056f86/thanos-query/0.log"
Jul 07 17:35:54.762249 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.762226    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79c9458997-rrnwn_17f6c6b7-c8eb-4101-bdcc-bde59e056f86/kube-rbac-proxy-web/0.log"
Jul 07 17:35:54.787372 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.787350    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79c9458997-rrnwn_17f6c6b7-c8eb-4101-bdcc-bde59e056f86/kube-rbac-proxy/0.log"
Jul 07 17:35:54.807903 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.807871    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79c9458997-rrnwn_17f6c6b7-c8eb-4101-bdcc-bde59e056f86/prom-label-proxy/0.log"
Jul 07 17:35:54.831379 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.831359    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79c9458997-rrnwn_17f6c6b7-c8eb-4101-bdcc-bde59e056f86/kube-rbac-proxy-rules/0.log"
Jul 07 17:35:54.861791 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:54.861764    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79c9458997-rrnwn_17f6c6b7-c8eb-4101-bdcc-bde59e056f86/kube-rbac-proxy-metrics/0.log"
Jul 07 17:35:56.860840 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:56.860811    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-fjddn_232694f9-b34b-49bd-b61a-e72e39479653/download-server/0.log"
Jul 07 17:35:57.073361 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073321    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"]
Jul 07 17:35:57.073714 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073695    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container"
Jul 07 17:35:57.073800 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073715    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container"
Jul 07 17:35:57.073800 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073734    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kube-rbac-proxy"
Jul 07 17:35:57.073800 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073742    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kube-rbac-proxy"
Jul 07 17:35:57.073800 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073761    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kube-rbac-proxy"
Jul 07 17:35:57.073800 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073770    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kube-rbac-proxy"
Jul 07 17:35:57.073800 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073779    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container"
Jul 07 17:35:57.073800 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073788    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container"
Jul 07 17:35:57.073800 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073802    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kube-rbac-proxy"
Jul 07 17:35:57.074217 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073810    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kube-rbac-proxy"
Jul 07 17:35:57.074217 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073822    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container"
Jul 07 17:35:57.074217 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073830    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container"
Jul 07 17:35:57.074217 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073901    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kserve-container"
Jul 07 17:35:57.074217 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073913    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kserve-container"
Jul 07 17:35:57.074217 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073921    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kserve-container"
Jul 07 17:35:57.074217 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073930    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d79b682-bdc2-4c76-9a46-18087eda25d0" containerName="kube-rbac-proxy"
Jul 07 17:35:57.074217 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073941    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a4f78b-fe8f-47b4-a95f-560ee160bbea" containerName="kube-rbac-proxy"
Jul 07 17:35:57.074217 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.073951    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e8ca15-e9b2-4ee0-bc59-46cde78e5796" containerName="kube-rbac-proxy"
Jul 07 17:35:57.077096 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.077075    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.079450 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.079432    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ppjh7\"/\"kube-root-ca.crt\""
Jul 07 17:35:57.079550 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.079428    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ppjh7\"/\"openshift-service-ca.crt\""
Jul 07 17:35:57.080134 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.080118    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ppjh7\"/\"default-dockercfg-lc92d\""
Jul 07 17:35:57.085815 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.085793    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"]
Jul 07 17:35:57.163274 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.163207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfwg9\" (UniqueName: \"kubernetes.io/projected/930abbbd-b65a-4a67-ad98-136d0be08363-kube-api-access-gfwg9\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.163274 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.163238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-podres\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.163453 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.163301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-proc\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.163453 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.163333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-lib-modules\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.163453 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.163372    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-sys\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.264341 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.264302    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-proc\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.264341 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.264344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-lib-modules\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.264509 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.264382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-sys\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.264509 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.264419    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-proc\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.264509 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.264475    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-sys\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.264509 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.264485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfwg9\" (UniqueName: \"kubernetes.io/projected/930abbbd-b65a-4a67-ad98-136d0be08363-kube-api-access-gfwg9\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.264637 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.264514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-podres\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.264637 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.264537    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-lib-modules\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.264637 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.264600    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/930abbbd-b65a-4a67-ad98-136d0be08363-podres\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.272581 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.272555    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfwg9\" (UniqueName: \"kubernetes.io/projected/930abbbd-b65a-4a67-ad98-136d0be08363-kube-api-access-gfwg9\") pod \"perf-node-gather-daemonset-qvkm5\" (UID: \"930abbbd-b65a-4a67-ad98-136d0be08363\") " pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.387834 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.387807    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.508915 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.508883    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"]
Jul 07 17:35:57.512008 ip-10-0-142-63 kubenswrapper[2559]: W0707 17:35:57.511968    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod930abbbd_b65a_4a67_ad98_136d0be08363.slice/crio-b65c285a13f9c2a521aa4457ad6c0f89a97cb6b7c3ad7cbc403be47559c6ed73 WatchSource:0}: Error finding container b65c285a13f9c2a521aa4457ad6c0f89a97cb6b7c3ad7cbc403be47559c6ed73: Status 404 returned error can't find the container with id b65c285a13f9c2a521aa4457ad6c0f89a97cb6b7c3ad7cbc403be47559c6ed73
Jul 07 17:35:57.809146 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.809111    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5" event={"ID":"930abbbd-b65a-4a67-ad98-136d0be08363","Type":"ContainerStarted","Data":"b4c9da29bf9daae00c9a2a2095150bc346ec891ec886258d0b562d59c2adf4e4"}
Jul 07 17:35:57.809346 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.809160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5" event={"ID":"930abbbd-b65a-4a67-ad98-136d0be08363","Type":"ContainerStarted","Data":"b65c285a13f9c2a521aa4457ad6c0f89a97cb6b7c3ad7cbc403be47559c6ed73"}
Jul 07 17:35:57.809346 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.809235    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:35:57.824581 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:57.824536    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5" podStartSLOduration=0.824523083 podStartE2EDuration="824.523083ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:35:57.822519377 +0000 UTC m=+2896.086056010" watchObservedRunningTime="2026-07-07 17:35:57.824523083 +0000 UTC m=+2896.088059736"
Jul 07 17:35:58.230894 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:58.230807    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4lhzc_5409188f-4647-4517-a322-06c705e56fff/dns/0.log"
Jul 07 17:35:58.250893 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:58.250868    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4lhzc_5409188f-4647-4517-a322-06c705e56fff/kube-rbac-proxy/0.log"
Jul 07 17:35:58.406652 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:58.406621    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tvtzh_67a4be0d-4845-43ec-a5d3-2952c9a6d01f/dns-node-resolver/0.log"
Jul 07 17:35:58.823110 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:58.823079    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d8f87c964-kkjnq_1321693f-e32d-4e48-9754-bceb0ec4431d/registry/0.log"
Jul 07 17:35:58.863325 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:35:58.863304    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ftff6_4a1625b5-d332-4801-8459-07c873619a65/node-ca/0.log"
Jul 07 17:36:00.049171 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:00.049144    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-twsng_1b7c6701-1f14-4950-9d40-25e7c24fd3b4/serve-healthcheck-canary/0.log"
Jul 07 17:36:00.477136 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:00.477063    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cwz47_f92b530c-2cf9-4e35-81a8-be8db3daaf48/kube-rbac-proxy/0.log"
Jul 07 17:36:00.499369 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:00.499342    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cwz47_f92b530c-2cf9-4e35-81a8-be8db3daaf48/exporter/0.log"
Jul 07 17:36:00.523521 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:00.523491    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cwz47_f92b530c-2cf9-4e35-81a8-be8db3daaf48/extractor/0.log"
Jul 07 17:36:02.632539 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:02.632502    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-85768989b-2g66c_e2ea16fc-9a62-44dc-8ceb-7819773890f9/manager/0.log"
Jul 07 17:36:02.652320 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:02.652298    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-92wk6_1919256d-0c4b-4f30-9a3c-63c9bde7d35b/manager/0.log"
Jul 07 17:36:02.757887 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:02.757862    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-zp428_bcc00f23-8c4e-46c2-a660-010ec659be5c/manager/0.log"
Jul 07 17:36:02.834826 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:02.834797    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-5v7z7_dc7fb89e-0a70-4cf4-a91a-76520ee41aa5/seaweedfs/0.log"
Jul 07 17:36:03.825831 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:03.825802    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ppjh7/perf-node-gather-daemonset-qvkm5"
Jul 07 17:36:07.773204 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:07.773174    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-64nzh_b9787f13-a9f7-4e87-9051-2c50eeb65db0/kube-multus/0.log"
Jul 07 17:36:07.956127 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:07.956061    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg2f8_709bde3f-4f89-4234-8ec7-c96fcb8333f5/kube-multus-additional-cni-plugins/0.log"
Jul 07 17:36:07.979234 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:07.979211    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg2f8_709bde3f-4f89-4234-8ec7-c96fcb8333f5/egress-router-binary-copy/0.log"
Jul 07 17:36:08.002444 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:08.002425    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg2f8_709bde3f-4f89-4234-8ec7-c96fcb8333f5/cni-plugins/0.log"
Jul 07 17:36:08.021945 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:08.021924    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg2f8_709bde3f-4f89-4234-8ec7-c96fcb8333f5/bond-cni-plugin/0.log"
Jul 07 17:36:08.043377 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:08.043361    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg2f8_709bde3f-4f89-4234-8ec7-c96fcb8333f5/routeoverride-cni/0.log"
Jul 07 17:36:08.063534 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:08.063516    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg2f8_709bde3f-4f89-4234-8ec7-c96fcb8333f5/whereabouts-cni-bincopy/0.log"
Jul 07 17:36:08.084439 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:08.084421    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg2f8_709bde3f-4f89-4234-8ec7-c96fcb8333f5/whereabouts-cni/0.log"
Jul 07 17:36:08.414464 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:08.414436    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vp8v6_fb7c2897-e46e-456b-b3d2-7f419224c6b3/network-metrics-daemon/0.log"
Jul 07 17:36:08.434416 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:08.434396    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vp8v6_fb7c2897-e46e-456b-b3d2-7f419224c6b3/kube-rbac-proxy/0.log"
Jul 07 17:36:09.531589 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:09.531564    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkhn9_282109b7-c9be-4d64-a867-4281aa2b16b9/ovn-controller/0.log"
Jul 07 17:36:09.548949 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:09.548924    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkhn9_282109b7-c9be-4d64-a867-4281aa2b16b9/ovn-acl-logging/0.log"
Jul 07 17:36:09.570583 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:09.570557    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkhn9_282109b7-c9be-4d64-a867-4281aa2b16b9/kube-rbac-proxy-node/0.log"
Jul 07 17:36:09.591123 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:09.591090    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkhn9_282109b7-c9be-4d64-a867-4281aa2b16b9/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 17:36:09.613805 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:09.613777    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkhn9_282109b7-c9be-4d64-a867-4281aa2b16b9/northd/0.log"
Jul 07 17:36:09.636584 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:09.636560    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkhn9_282109b7-c9be-4d64-a867-4281aa2b16b9/nbdb/0.log"
Jul 07 17:36:09.660317 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:09.660294    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkhn9_282109b7-c9be-4d64-a867-4281aa2b16b9/sbdb/0.log"
Jul 07 17:36:09.703021 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:09.702997    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkhn9_282109b7-c9be-4d64-a867-4281aa2b16b9/ovnkube-controller/0.log"
Jul 07 17:36:10.595606 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:10.595572    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-pjtgp_077f2741-19ab-4827-ace5-c11203efc785/network-check-target-container/0.log"
Jul 07 17:36:11.531652 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:11.531624    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-m5zt9_da05c267-ecba-4f27-883a-f5f9792142a0/iptables-alerter/0.log"
Jul 07 17:36:12.155259 ip-10-0-142-63 kubenswrapper[2559]: I0707 17:36:12.155231    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-kz8s2_7b6ad925-dd67-4e6b-8bdc-98c37416d65f/tuned/0.log"