Jul 09 18:02:08.321985 ip-10-0-135-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 18:02:08.321996 ip-10-0-135-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 18:02:08.322004 ip-10-0-135-87 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 18:02:08.322217 ip-10-0-135-87 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 18:02:18.418051 ip-10-0-135-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 18:02:18.418066 ip-10-0-135-87 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7ecf02821f6a42f19af4a37daf96cbe8 --
Jul 09 18:04:15.735726 ip-10-0-135-87 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:04:16.192275 ip-10-0-135-87 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.126533    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131530    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131544    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131549    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131553    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131558    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131562    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131565    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131569    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:16.200624 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131573    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131576    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131579    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131581    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131584    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131587    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131589    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131593    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131595    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131598    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131602    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131604    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131607    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131610    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131613    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131615    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131618    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131620    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131623    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131625    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:16.201544 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131628    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131631    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131633    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131636    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131638    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131641    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131644    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131647    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131649    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131652    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131655    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131661    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131666    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131671    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131675    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131678    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131680    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131683    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131687    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:16.202150 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131690    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131693    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131697    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131700    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131703    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131705    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131708    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131712    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131715    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131718    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131720    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131723    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131725    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131728    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131730    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131733    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131735    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131737    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131740    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131742    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:16.202771 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131745    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131748    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131750    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131753    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131757    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131760    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131763    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131765    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131768    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131771    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131773    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131776    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131778    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131781    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131783    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131786    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131788    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131791    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131794    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131796    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:16.203353 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131799    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131803    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131805    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131808    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131810    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131813    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131815    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131818    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131820    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131823    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131826    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.131828    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132530    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132537    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132540    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132544    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132549    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132552    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132554    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132558    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:16.203929 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132561    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132563    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132567    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132571    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132574    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132578    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132581    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132584    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132587    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132590    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132594    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132596    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132599    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132602    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132604    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132606    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132609    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132611    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132614    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:16.204740 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132617    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132619    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132622    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132624    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132627    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132629    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132632    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132635    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132637    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132641    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132644    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132646    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132649    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132652    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132655    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132658    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132660    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132663    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132665    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:16.205721 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132668    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132670    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132673    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132676    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132679    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132681    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132684    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132687    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132689    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132692    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132694    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132697    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132699    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132702    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132704    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132710    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132713    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132716    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132719    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132721    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:16.206663 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132724    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132727    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132731    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132733    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132736    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132738    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132741    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132744    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132747    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132749    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132752    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132754    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132757    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132760    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132762    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132765    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132767    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132770    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132772    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132775    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:16.207339 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132777    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132780    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132782    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132785    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132787    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132790    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132792    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132794    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132797    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132799    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132802    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132804    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.132806    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133146    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133151    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133154    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133158    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133160    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133163    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:16.208013 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133166    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133170    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133174    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133177    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133180    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133182    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133186    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133188    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133191    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133194    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133196    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133199    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133202    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133204    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133207    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133210    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133212    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133215    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133218    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:16.208898 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133220    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133223    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133225    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133228    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133230    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133232    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133235    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133237    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133241    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133244    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133247    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133249    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133252    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133254    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133257    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133260    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133265    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133268    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133271    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133273    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:16.425340 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133277    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133279    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133282    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133284    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133287    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133289    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133292    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133295    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133298    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133301    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133303    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133306    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133309    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133311    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133313    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133316    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133318    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133320    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133323    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133325    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:16.443807 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133329    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133333    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133336    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133340    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133342    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133345    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133347    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133350    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133353    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133356    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133358    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133361    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133363    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133366    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133368    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133371    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133373    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133376    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133378    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:16.444629 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133380    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133384    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133386    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133389    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133391    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133394    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133396    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133399    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133401    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133404    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133406    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133409    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133411    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133415    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.133417    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133509    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133521    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133527    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133532    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133535    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133539    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133543    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:04:16.445462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133547    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133551    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133553    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133557    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133560    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133563    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133566    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133569    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133572    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133574    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133577    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133583    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133586    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133589    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133592    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133595    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133599    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133602    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133605    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133608    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133611    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133614    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133617    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133622    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:04:16.446226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133625    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133629    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133632    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133635    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133638    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133640    2560 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133644    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133648    2560 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133652    2560 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133654    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133657    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133660    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133664    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133666    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133670    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133673    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133675    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133678    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133681    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133685    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133688    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133690    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133693    2560 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133696    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133700    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:04:16.447050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133703    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133706    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133709    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133712    2560 flags.go:64] FLAG: --help="false"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133714    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133717    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133721    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133724    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133727    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133730    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133733    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133736    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133739    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133742    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133744    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133747    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133753    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133756    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133759    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133762    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133765    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133767    2560 flags.go:64] FLAG: --lock-file=""
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133770    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133773    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:04:16.447935 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133776    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133783    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133788    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133790    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133793    2560 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133796    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133799    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133802    2560 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133804    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133808    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133811    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133815    2560 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133818    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133821    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133826    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133829    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133832    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133835    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133838    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133845    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133848    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133850    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133853    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:04:16.448738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133856    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133861    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133864    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133867    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133870    2560 flags.go:64] FLAG: --port="10250"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133873    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133876    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b828ffa663a6f454"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133879    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133882    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133885    2560 flags.go:64] FLAG: --register-node="true"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133887    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133892    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133895    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133898    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133901    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133904    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133907    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133910    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133913    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133916    2560 flags.go:64] FLAG: --runonce="false"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133919    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133922    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133925    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133927    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133931    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133934    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:04:16.449883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133937    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133940    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133943    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133946    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133949    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133952    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133955    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133958    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133960    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133967    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133970    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133972    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133976    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133979    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133982    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133985    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133987    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133992    2560 flags.go:64] FLAG: --v="2"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133996    2560 flags.go:64] FLAG: --version="false"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.133999    2560 flags.go:64] FLAG: --vmodule=""
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.134003    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.134006    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134112    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134115    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134118    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:16.450696 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134121    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134124    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134126    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134129    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134132    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134136    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134139    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134141    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134144    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134148    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134151    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134154    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134156    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134159    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134161    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134164    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134167    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134170    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134173    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:16.681141 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134175    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134178    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134180    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134183    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134185    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134189    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134192    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134195    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134197    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134200    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134202    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134205    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134208    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134210    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134213    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134215    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134218    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134220    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134224    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134227    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:16.682115 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134229    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134232    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134235    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134237    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134240    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134242    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134245    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134247    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134250    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134252    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134255    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134258    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134261    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134263    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134266    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134268    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134271    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134274    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134277    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134280    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:16.682795 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134282    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134285    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134287    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134290    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134292    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134295    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134297    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134299    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134302    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134305    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134310    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134312    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134315    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134317    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134320    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134322    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134326    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134329    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134332    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:16.683412 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134337    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134340    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134343    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134346    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134349    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134352    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134355    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134358    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134360    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134363    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134367    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134370    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134372    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134375    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134378    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134380    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134383    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:16.683977 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.134385    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.134972    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.142412    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.142426    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.142460    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.142467    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142548    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142556    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142559    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142562    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142565    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142568    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142572    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142576    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:16.684540 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142579    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142582    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142585    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142587    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142591    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142595    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142598    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142601    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142604    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142607    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142610    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142614    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142617    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142620    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142622    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142625    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142628    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142630    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142633    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:16.684964 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142636    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142639    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142641    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142644    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142646    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142649    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142653    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142656    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142658    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142661    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142663    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142666    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142668    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142672    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142675    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142678    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142681    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142684    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142687    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142696    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:16.685538 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142699    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142701    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142704    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142707    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142711    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142713    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142716    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142719    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142721    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142723    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142726    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142729    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142731    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142734    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142736    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142739    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142742    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142744    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142748    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142750    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:16.686122 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142753    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142755    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142758    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142761    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142763    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142766    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142768    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142771    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142773    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142776    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142778    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142781    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142789    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142792    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142795    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142797    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142801    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142803    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142806    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142809    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:16.686810 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142811    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142814    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142817    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142820    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142823    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142825    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142827    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142830    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142832    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142835    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142838    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142841    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.142846    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142980    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142985    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:16.687372 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142988    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142991    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142993    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142996    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.142999    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143001    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143004    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143006    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143009    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143011    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143020    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143022    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143025    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143029    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143032    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143034    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143037    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143039    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143042    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143045    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:16.687906 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143047    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143050    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143053    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143055    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143058    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143060    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143063    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143065    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143069    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143072    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143074    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143077    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143079    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143082    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143084    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143087    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143089    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143092    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143094    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143097    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:16.688470 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143099    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143102    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143104    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143114    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143117    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143122    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143124    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143127    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143129    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143132    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143134    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143137    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143139    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143142    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143144    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143147    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143149    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143152    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143154    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143157    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:16.689094 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143161    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143164    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143166    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143169    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143171    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143174    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143177    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143179    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143183    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143186    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143189    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143192    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143195    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143197    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143201    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143204    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143213    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143218    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143221    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:16.689749 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143224    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143227    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143230    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143233    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143236    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143238    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143241    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143243    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143246    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143248    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143251    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143253    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143256    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143259    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143262    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143264    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143267    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:16.690329 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:16.143269    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.143274    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.143405    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.145865    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.147236    2560 server.go:1019] "Starting client certificate rotation"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.147331    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.147362    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.149632    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.150760    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.161413    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.167125    2560 log.go:25] "Validated CRI v1 image API"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.167935    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:04:16.690874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.170601    2560 fs.go:135] Filesystem UUIDs: map[1ff38d84-dc34-42c7-94f3-028d90317ef0:/dev/nvme0n1p4 58e044b1-0a8d-48ee-b774-9ffaa80bec6e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 09 18:04:16.691289 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.170625    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:04:16.691289 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.170953    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.175446    2560 manager.go:216] Machine: {Timestamp:2026-07-09 18:04:16.174651929 +0000 UTC m=+0.326289197 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3114842 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b9e774c10fe8005d7a13793ebb0b4 SystemUUID:ec2b9e77-4c10-fe80-05d7-a13793ebb0b4 BootID:7ecf0282-1f6a-42f1-9af4-a37daf96cbe8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0f:99:04:32:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0f:99:04:32:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:11:72:95:ea:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.175550    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.175647    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.177492    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.177517    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-87.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.178747    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.178758    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.178765    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.179341    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.180445    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.183244    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.183349    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.183374    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.183383    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.183400    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.183409    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.183417    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:04:16.691469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.183757    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.183772    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.185958    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.187570    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.187597    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.187728    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.187739    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.187745    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.187751    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.187759    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.187764    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.188364    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.188369    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.188375    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.188380    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.188388    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.188397    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.188414    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.188419    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.191296    2560 server.go:1268] "Started kubelet"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.191527    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.191789    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.192457    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.192728    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.194724    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.195587    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.195613    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.195675    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-87.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.197237    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.197994    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-87.ec2.internal.18c0b12c4878d6f7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-87.ec2.internal,UID:ip-10-0-135-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-87.ec2.internal,},FirstTimestamp:2026-07-09 18:04:16.191272695 +0000 UTC m=+0.342909957,LastTimestamp:2026-07-09 18:04:16.191272695 +0000 UTC m=+0.342909957,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-87.ec2.internal,}"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.203022    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r962b"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.204114    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.204132    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.204243    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.204615    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.204627    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.204644    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.204750    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.204760    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.205000    2560 factory.go:55] Registering systemd factory
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.205015    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:04:16.692350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.205227    2560 factory.go:153] Registering CRI-O factory
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.205240    2560 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.205291    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.205331    2560 factory.go:103] Registering Raw factory
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.205347    2560 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.205886    2560 manager.go:349] Starting recovery of all containers
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.207249    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.207353    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.215375    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r962b"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.224006    2560 manager.go:354] Recovery completed
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.228006    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.229728    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.229751    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.229760    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.230228    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.230247    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.230262    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.230272    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.230951    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.230976    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.230991    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.231286    2560 policy_none.go:49] "None policy: Start"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.231299    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.231308    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.232261    2560 policy_none.go:47] "Start"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.235411    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-87.ec2.internal.18c0b12c4ac3c667  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-87.ec2.internal,UID:ip-10-0-135-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-87.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-87.ec2.internal,},FirstTimestamp:2026-07-09 18:04:16.229738087 +0000 UTC m=+0.381375353,LastTimestamp:2026-07-09 18:04:16.229738087 +0000 UTC m=+0.381375353,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-87.ec2.internal,}"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.267678    2560 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.267793    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.267802    2560 server.go:87] "Starting device plugin registration server"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.267980    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.267989    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.268078    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.268148    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.268157    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.268605    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.268647    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.368999    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.369736    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:16.949469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.369762    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.369774    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.369806    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.377294    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.377313    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-87.ec2.internal\": node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.381197    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.382394    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.382429    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.382447    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.382479    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.385000    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.395625    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.483388    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal"]
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.483472    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.484347    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.484380    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.484399    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.485753    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.485872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.485906    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.486483    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.486498    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.486526    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.486542    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.486529    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.486632    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.487757    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.487782    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.488398    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.488421    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.488434    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.496343    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.508735    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f2dcd2c4647c8a5c8152fee259ad6a9-config\") pod \"kube-apiserver-proxy-ip-10-0-135-87.ec2.internal\" (UID: \"6f2dcd2c4647c8a5c8152fee259ad6a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.508765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/291def8d0af3fe3d1b87f627e9c4338e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal\" (UID: \"291def8d0af3fe3d1b87f627e9c4338e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.508803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/291def8d0af3fe3d1b87f627e9c4338e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal\" (UID: \"291def8d0af3fe3d1b87f627e9c4338e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.952111 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.517227    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-87.ec2.internal\" not found" node="ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.520667    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-87.ec2.internal\" not found" node="ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.596494    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.609498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/291def8d0af3fe3d1b87f627e9c4338e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal\" (UID: \"291def8d0af3fe3d1b87f627e9c4338e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.609524    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/291def8d0af3fe3d1b87f627e9c4338e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal\" (UID: \"291def8d0af3fe3d1b87f627e9c4338e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.609571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f2dcd2c4647c8a5c8152fee259ad6a9-config\") pod \"kube-apiserver-proxy-ip-10-0-135-87.ec2.internal\" (UID: \"6f2dcd2c4647c8a5c8152fee259ad6a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.609600    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f2dcd2c4647c8a5c8152fee259ad6a9-config\") pod \"kube-apiserver-proxy-ip-10-0-135-87.ec2.internal\" (UID: \"6f2dcd2c4647c8a5c8152fee259ad6a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.609604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/291def8d0af3fe3d1b87f627e9c4338e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal\" (UID: \"291def8d0af3fe3d1b87f627e9c4338e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.609633    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/291def8d0af3fe3d1b87f627e9c4338e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal\" (UID: \"291def8d0af3fe3d1b87f627e9c4338e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.696724    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.803244    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.818745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:16.822857    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:16.953173 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:16.906273    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:17.209193 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:17.007444    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:17.209193 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:17.107815    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:17.209193 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.147108    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:04:17.209193 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.147265    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:17.209193 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.173244    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:17.209193 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:17.208013    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:17.418378 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.219062    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:59:16 +0000 UTC" deadline="2028-04-18 09:15:29.019929528 +0000 UTC"
Jul 09 18:04:17.418378 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.219082    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15567h11m11.800850708s"
Jul 09 18:04:17.418378 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:17.308136    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:17.418378 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:17.408517    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-87.ec2.internal\" not found"
Jul 09 18:04:17.420151 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.420042    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.421808    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.421841    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.421853    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.431221    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.505527    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.516984    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.518068    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal"
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.535129    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:17.612497    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291def8d0af3fe3d1b87f627e9c4338e.slice/crio-9a0cd548e3a21559f7dd0ecf461f7070f8d0531b9d756bb5183e349eee087952 WatchSource:0}: Error finding container 9a0cd548e3a21559f7dd0ecf461f7070f8d0531b9d756bb5183e349eee087952: Status 404 returned error can't find the container with id 9a0cd548e3a21559f7dd0ecf461f7070f8d0531b9d756bb5183e349eee087952
Jul 09 18:04:17.616359 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:17.612917    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2dcd2c4647c8a5c8152fee259ad6a9.slice/crio-f9c827e2077c4d26c3819d2e521dd50828ac9f06b30820ac5a9b92de6a932c97 WatchSource:0}: Error finding container f9c827e2077c4d26c3819d2e521dd50828ac9f06b30820ac5a9b92de6a932c97: Status 404 returned error can't find the container with id f9c827e2077c4d26c3819d2e521dd50828ac9f06b30820ac5a9b92de6a932c97
Jul 09 18:04:17.619678 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.618448    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:04:17.855293 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:17.688940    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:18.095204 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.027039    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.184370    2560 apiserver.go:52] "Watching apiserver"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.193186    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.195735    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-zsbvk","openshift-network-diagnostics/network-check-target-gbg4t","kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp","openshift-cluster-node-tuning-operator/tuned-rvc7q","openshift-dns/node-resolver-5xkmx","openshift-multus/multus-mm9zg","openshift-network-operator/iptables-alerter-75tj7","openshift-ovn-kubernetes/ovnkube-node-p7npv","kube-system/global-pull-secret-syncer-5wc7j","kube-system/konnectivity-agent-8hb8n","openshift-image-registry/node-ca-f6mv8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal","openshift-multus/multus-additional-cni-plugins-7gzr4"]
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.198471    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.198566    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.200828    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.203149    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.204242    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.204516    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.204587    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.204629    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.205420    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.205670    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.205712    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dm6h4\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.205890    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.206069    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h7sj8\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.206502    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.209114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.210737    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.210793    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.210978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-67zj2\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.211123    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.211574    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.214728    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-run-netns\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.214764    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg5lq\" (UniqueName: \"kubernetes.io/projected/8220436b-a8e2-4f0e-ab53-776cc9660d6a-kube-api-access-rg5lq\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.214792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-device-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.214818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-cnibin\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.232813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.214840    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.214842    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-os-release\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.214896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-var-lib-kubelet\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.214937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-daemon-config\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.214971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-run-multus-certs\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-etc-kubernetes\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215057    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-etc-selinux\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcj9\" (UniqueName: \"kubernetes.io/projected/1803cc34-6cae-4580-a034-b67c1cab8b8e-kube-api-access-pbcj9\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb158c6c-ad86-4462-a1e0-fe975d0055c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215137    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-os-release\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8220436b-a8e2-4f0e-ab53-776cc9660d6a-cni-binary-copy\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-var-lib-cni-bin\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-hostroot\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215260    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbzp8\" (UniqueName: \"kubernetes.io/projected/22f19aa3-2b80-4581-b937-4abbb0162b51-kube-api-access-jbzp8\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215311    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:18.236326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-conf-dir\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-registration-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb158c6c-ad86-4462-a1e0-fe975d0055c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-system-cni-dir\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.215548    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-system-cni-dir\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-cni-dir\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-run-k8s-cni-cncf-io\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-var-lib-cni-multus\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-socket-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb158c6c-ad86-4462-a1e0-fe975d0055c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216743    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-socket-dir-parent\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-sys-fs\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxnqc\" (UniqueName: \"kubernetes.io/projected/bb158c6c-ad86-4462-a1e0-fe975d0055c4-kube-api-access-mxnqc\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.216811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-cnibin\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.220312    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.220523    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.222071    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2mpbg\""
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.222093    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:18.237347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.222188    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vc79s\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.225111    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.227668    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.228809    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.228849    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.228861    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rp9v2\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.229285    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.233034    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.236118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.236273    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.237635    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.237677    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.237888    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.237904    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.238027    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:04:18.238389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.238158    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:18.239358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.238621    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rhh5w\""
Jul 09 18:04:18.239358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.238662    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:04:18.242213 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.240810    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.244291 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.242312    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:04:18.244291 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.242537    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-648ww\""
Jul 09 18:04:18.244291 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.242629    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:04:18.244291 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.243224    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:04:18.244291 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.243947    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xfc58\""
Jul 09 18:04:18.244291 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.244004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:18.244291 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.244092    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:18.244291 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.244198    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:04:18.245938 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.244309    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:04:18.257407 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.255920    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q8qc2"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.267997    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q8qc2"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.305827    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-tuned\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-log-socket\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxnqc\" (UniqueName: \"kubernetes.io/projected/bb158c6c-ad86-4462-a1e0-fe975d0055c4-kube-api-access-mxnqc\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-cnibin\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-run-netns\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg5lq\" (UniqueName: \"kubernetes.io/projected/8220436b-a8e2-4f0e-ab53-776cc9660d6a-kube-api-access-rg5lq\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-cnibin\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-run-netns\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-device-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-device-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8cv8\" (UniqueName: \"kubernetes.io/projected/9afa7d8f-5b83-494b-9166-c8fd3fb1282b-kube-api-access-x8cv8\") pod \"node-resolver-5xkmx\" (UID: \"9afa7d8f-5b83-494b-9166-c8fd3fb1282b\") " pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317477    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-kubelet\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-os-release\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-daemon-config\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-run-multus-certs\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.338812 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-etc-selinux\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317674    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f4918fd2-0cae-42eb-b8dc-855565f0298b-ovnkube-config\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317681    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-run-multus-certs\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-os-release\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbzp8\" (UniqueName: \"kubernetes.io/projected/22f19aa3-2b80-4581-b937-4abbb0162b51-kube-api-access-jbzp8\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.317817    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-etc-selinux\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.317911    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318120    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-daemon-config\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-registration-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb158c6c-ad86-4462-a1e0-fe975d0055c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318316    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-registration-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.318329    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs podName:1803cc34-6cae-4580-a034-b67c1cab8b8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:18.818291601 +0000 UTC m=+2.969928866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs") pod "network-metrics-daemon-zsbvk" (UID: "1803cc34-6cae-4580-a034-b67c1cab8b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:18.340185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqs5q\" (UniqueName: \"kubernetes.io/projected/0265afbf-1a11-438e-b093-11f6dcfbd075-kube-api-access-cqs5q\") pod \"iptables-alerter-75tj7\" (UID: \"0265afbf-1a11-438e-b093-11f6dcfbd075\") " pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-sysctl-conf\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-lib-modules\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318459    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-etc-openvswitch\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-run-ovn\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-system-cni-dir\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-cni-dir\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9-konnectivity-ca\") pod \"konnectivity-agent-8hb8n\" (UID: \"b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9\") " pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-system-cni-dir\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c33e40f4-ff09-4aa9-9fb1-7c8de50e683a-host\") pod \"node-ca-f6mv8\" (UID: \"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a\") " pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4516b8-9d8d-48ab-b802-794ad1ed6874-tmp\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9afa7d8f-5b83-494b-9166-c8fd3fb1282b-hosts-file\") pod \"node-resolver-5xkmx\" (UID: \"9afa7d8f-5b83-494b-9166-c8fd3fb1282b\") " pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318735    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-systemd-units\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318760    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2pq\" (UniqueName: \"kubernetes.io/projected/c33e40f4-ff09-4aa9-9fb1-7c8de50e683a-kube-api-access-8g2pq\") pod \"node-ca-f6mv8\" (UID: \"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a\") " pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-systemd\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-cni-dir\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.341358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb158c6c-ad86-4462-a1e0-fe975d0055c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb158c6c-ad86-4462-a1e0-fe975d0055c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-socket-dir-parent\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318927    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-run-openvswitch\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-kubelet-config\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.318980    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-run-netns\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319001    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-socket-dir-parent\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-cni-bin\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-cni-netd\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-cnibin\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-var-lib-kubelet\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-etc-kubernetes\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-sysctl-d\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-cnibin\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnlz\" (UniqueName: \"kubernetes.io/projected/f4918fd2-0cae-42eb-b8dc-855565f0298b-kube-api-access-btnlz\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcj9\" (UniqueName: \"kubernetes.io/projected/1803cc34-6cae-4580-a034-b67c1cab8b8e-kube-api-access-pbcj9\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-var-lib-kubelet\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.342996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319192    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb158c6c-ad86-4462-a1e0-fe975d0055c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-etc-kubernetes\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-os-release\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb158c6c-ad86-4462-a1e0-fe975d0055c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8220436b-a8e2-4f0e-ab53-776cc9660d6a-cni-binary-copy\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-var-lib-cni-bin\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-os-release\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-hostroot\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319371    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-hostroot\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319402    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-modprobe-d\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-var-lib-cni-bin\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319429    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-conf-dir\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-dbus\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319477    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-sys\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319481    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-multus-conf-dir\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-var-lib-kubelet\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-run-systemd\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f4918fd2-0cae-42eb-b8dc-855565f0298b-ovnkube-script-lib\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.344129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-system-cni-dir\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-kubernetes\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319638    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-host\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-node-log\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb158c6c-ad86-4462-a1e0-fe975d0055c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8220436b-a8e2-4f0e-ab53-776cc9660d6a-cni-binary-copy\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319744    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-system-cni-dir\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319843    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319872    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f4918fd2-0cae-42eb-b8dc-855565f0298b-env-overrides\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319916    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0265afbf-1a11-438e-b093-11f6dcfbd075-iptables-alerter-script\") pod \"iptables-alerter-75tj7\" (UID: \"0265afbf-1a11-438e-b093-11f6dcfbd075\") " pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c33e40f4-ff09-4aa9-9fb1-7c8de50e683a-serviceca\") pod \"node-ca-f6mv8\" (UID: \"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a\") " pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.319983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-slash\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-var-lib-openvswitch\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4918fd2-0cae-42eb-b8dc-855565f0298b-ovn-node-metrics-cert\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-run-k8s-cni-cncf-io\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-var-lib-cni-multus\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.345996 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-socket-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-sysconfig\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320203    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-run-k8s-cni-cncf-io\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqh2\" (UniqueName: \"kubernetes.io/projected/1b4516b8-9d8d-48ab-b802-794ad1ed6874-kube-api-access-lbqh2\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afa7d8f-5b83-494b-9166-c8fd3fb1282b-tmp-dir\") pod \"node-resolver-5xkmx\" (UID: \"9afa7d8f-5b83-494b-9166-c8fd3fb1282b\") " pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-sys-fs\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320318    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-socket-dir\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0265afbf-1a11-438e-b093-11f6dcfbd075-host-slash\") pod \"iptables-alerter-75tj7\" (UID: \"0265afbf-1a11-438e-b093-11f6dcfbd075\") " pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8220436b-a8e2-4f0e-ab53-776cc9660d6a-host-var-lib-cni-multus\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9-agent-certs\") pod \"konnectivity-agent-8hb8n\" (UID: \"b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9\") " pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-run\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320497    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22f19aa3-2b80-4581-b937-4abbb0162b51-sys-fs\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.320542    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb158c6c-ad86-4462-a1e0-fe975d0055c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.327968    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.332547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxnqc\" (UniqueName: \"kubernetes.io/projected/bb158c6c-ad86-4462-a1e0-fe975d0055c4-kube-api-access-mxnqc\") pod \"multus-additional-cni-plugins-7gzr4\" (UID: \"bb158c6c-ad86-4462-a1e0-fe975d0055c4\") " pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.333020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbzp8\" (UniqueName: \"kubernetes.io/projected/22f19aa3-2b80-4581-b937-4abbb0162b51-kube-api-access-jbzp8\") pod \"aws-ebs-csi-driver-node-qr5bp\" (UID: \"22f19aa3-2b80-4581-b937-4abbb0162b51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.347166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.339589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcj9\" (UniqueName: \"kubernetes.io/projected/1803cc34-6cae-4580-a034-b67c1cab8b8e-kube-api-access-pbcj9\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:18.349476 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.347867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg5lq\" (UniqueName: \"kubernetes.io/projected/8220436b-a8e2-4f0e-ab53-776cc9660d6a-kube-api-access-rg5lq\") pod \"multus-mm9zg\" (UID: \"8220436b-a8e2-4f0e-ab53-776cc9660d6a\") " pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.389851    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal" event={"ID":"6f2dcd2c4647c8a5c8152fee259ad6a9","Type":"ContainerStarted","Data":"f9c827e2077c4d26c3819d2e521dd50828ac9f06b30820ac5a9b92de6a932c97"}
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.393194    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal" event={"ID":"291def8d0af3fe3d1b87f627e9c4338e","Type":"ContainerStarted","Data":"9a0cd548e3a21559f7dd0ecf461f7070f8d0531b9d756bb5183e349eee087952"}
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9-konnectivity-ca\") pod \"konnectivity-agent-8hb8n\" (UID: \"b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9\") " pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c33e40f4-ff09-4aa9-9fb1-7c8de50e683a-host\") pod \"node-ca-f6mv8\" (UID: \"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a\") " pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4516b8-9d8d-48ab-b802-794ad1ed6874-tmp\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9afa7d8f-5b83-494b-9166-c8fd3fb1282b-hosts-file\") pod \"node-resolver-5xkmx\" (UID: \"9afa7d8f-5b83-494b-9166-c8fd3fb1282b\") " pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421216    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9afa7d8f-5b83-494b-9166-c8fd3fb1282b-hosts-file\") pod \"node-resolver-5xkmx\" (UID: \"9afa7d8f-5b83-494b-9166-c8fd3fb1282b\") " pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c33e40f4-ff09-4aa9-9fb1-7c8de50e683a-host\") pod \"node-ca-f6mv8\" (UID: \"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a\") " pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-systemd-units\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2pq\" (UniqueName: \"kubernetes.io/projected/c33e40f4-ff09-4aa9-9fb1-7c8de50e683a-kube-api-access-8g2pq\") pod \"node-ca-f6mv8\" (UID: \"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a\") " pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-systemd\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-systemd-units\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-run-openvswitch\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-kubelet-config\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421435    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-run-openvswitch\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.491279 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-kubelet-config\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-run-netns\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-cni-bin\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-systemd\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421520    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-run-netns\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421524    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-cni-netd\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-cni-bin\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421564    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-sysctl-d\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-cni-netd\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnlz\" (UniqueName: \"kubernetes.io/projected/f4918fd2-0cae-42eb-b8dc-855565f0298b-kube-api-access-btnlz\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-modprobe-d\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-dbus\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421661    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-sys\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-var-lib-kubelet\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-run-systemd\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-dbus\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f4918fd2-0cae-42eb-b8dc-855565f0298b-ovnkube-script-lib\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-modprobe-d\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.493160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-sysctl-d\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-sys\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421776    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-run-systemd\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421797    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-var-lib-kubelet\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-kubernetes\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421827    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-host\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-node-log\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-kubernetes\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f4918fd2-0cae-42eb-b8dc-855565f0298b-env-overrides\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0265afbf-1a11-438e-b093-11f6dcfbd075-iptables-alerter-script\") pod \"iptables-alerter-75tj7\" (UID: \"0265afbf-1a11-438e-b093-11f6dcfbd075\") " pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421913    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-node-log\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421924    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-host\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c33e40f4-ff09-4aa9-9fb1-7c8de50e683a-serviceca\") pod \"node-ca-f6mv8\" (UID: \"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a\") " pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.421997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-slash\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.494087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-var-lib-openvswitch\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4918fd2-0cae-42eb-b8dc-855565f0298b-ovn-node-metrics-cert\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-sysconfig\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqh2\" (UniqueName: \"kubernetes.io/projected/1b4516b8-9d8d-48ab-b802-794ad1ed6874-kube-api-access-lbqh2\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afa7d8f-5b83-494b-9166-c8fd3fb1282b-tmp-dir\") pod \"node-resolver-5xkmx\" (UID: \"9afa7d8f-5b83-494b-9166-c8fd3fb1282b\") " pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0265afbf-1a11-438e-b093-11f6dcfbd075-host-slash\") pod \"iptables-alerter-75tj7\" (UID: \"0265afbf-1a11-438e-b093-11f6dcfbd075\") " pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422184    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9-agent-certs\") pod \"konnectivity-agent-8hb8n\" (UID: \"b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9\") " pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-run\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-tuned\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-log-socket\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8cv8\" (UniqueName: \"kubernetes.io/projected/9afa7d8f-5b83-494b-9166-c8fd3fb1282b-kube-api-access-x8cv8\") pod \"node-resolver-5xkmx\" (UID: \"9afa7d8f-5b83-494b-9166-c8fd3fb1282b\") " pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f4918fd2-0cae-42eb-b8dc-855565f0298b-ovnkube-script-lib\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-kubelet\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f4918fd2-0cae-42eb-b8dc-855565f0298b-env-overrides\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422360    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f4918fd2-0cae-42eb-b8dc-855565f0298b-ovnkube-config\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c33e40f4-ff09-4aa9-9fb1-7c8de50e683a-serviceca\") pod \"node-ca-f6mv8\" (UID: \"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a\") " pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-var-lib-openvswitch\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.496164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-slash\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422452    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0265afbf-1a11-438e-b093-11f6dcfbd075-host-slash\") pod \"iptables-alerter-75tj7\" (UID: \"0265afbf-1a11-438e-b093-11f6dcfbd075\") " pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs5q\" (UniqueName: \"kubernetes.io/projected/0265afbf-1a11-438e-b093-11f6dcfbd075-kube-api-access-cqs5q\") pod \"iptables-alerter-75tj7\" (UID: \"0265afbf-1a11-438e-b093-11f6dcfbd075\") " pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422504    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0265afbf-1a11-438e-b093-11f6dcfbd075-iptables-alerter-script\") pod \"iptables-alerter-75tj7\" (UID: \"0265afbf-1a11-438e-b093-11f6dcfbd075\") " pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-sysctl-conf\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-run\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-lib-modules\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afa7d8f-5b83-494b-9166-c8fd3fb1282b-tmp-dir\") pod \"node-resolver-5xkmx\" (UID: \"9afa7d8f-5b83-494b-9166-c8fd3fb1282b\") " pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422601    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-etc-openvswitch\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-run-ovn\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-lib-modules\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422639    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-etc-openvswitch\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-host-kubelet\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-log-socket\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.422910    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-sysctl-conf\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.423148    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-sysconfig\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.423196    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:18.497318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.423204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f4918fd2-0cae-42eb-b8dc-855565f0298b-ovnkube-config\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.423255    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret podName:c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:18.923237783 +0000 UTC m=+3.074875039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret") pod "global-pull-secret-syncer-5wc7j" (UID: "c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.423200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f4918fd2-0cae-42eb-b8dc-855565f0298b-run-ovn\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.423665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9-konnectivity-ca\") pod \"konnectivity-agent-8hb8n\" (UID: \"b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9\") " pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.425184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b4516b8-9d8d-48ab-b802-794ad1ed6874-etc-tuned\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.425351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9-agent-certs\") pod \"konnectivity-agent-8hb8n\" (UID: \"b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9\") " pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.425835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4516b8-9d8d-48ab-b802-794ad1ed6874-tmp\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.426079    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4918fd2-0cae-42eb-b8dc-855565f0298b-ovn-node-metrics-cert\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.434464    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqh2\" (UniqueName: \"kubernetes.io/projected/1b4516b8-9d8d-48ab-b802-794ad1ed6874-kube-api-access-lbqh2\") pod \"tuned-rvc7q\" (UID: \"1b4516b8-9d8d-48ab-b802-794ad1ed6874\") " pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.438671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2pq\" (UniqueName: \"kubernetes.io/projected/c33e40f4-ff09-4aa9-9fb1-7c8de50e683a-kube-api-access-8g2pq\") pod \"node-ca-f6mv8\" (UID: \"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a\") " pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.442501    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.442528    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.442543    2560 projected.go:196] Error preparing data for projected volume kube-api-access-xknv5 for pod openshift-network-diagnostics/network-check-target-gbg4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.442648    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5 podName:882b7841-7d01-46cb-afca-f21a77389624 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:18.942628014 +0000 UTC m=+3.094265285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xknv5" (UniqueName: "kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5") pod "network-check-target-gbg4t" (UID: "882b7841-7d01-46cb-afca-f21a77389624") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.445436    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8cv8\" (UniqueName: \"kubernetes.io/projected/9afa7d8f-5b83-494b-9166-c8fd3fb1282b-kube-api-access-x8cv8\") pod \"node-resolver-5xkmx\" (UID: \"9afa7d8f-5b83-494b-9166-c8fd3fb1282b\") " pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.446006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnlz\" (UniqueName: \"kubernetes.io/projected/f4918fd2-0cae-42eb-b8dc-855565f0298b-kube-api-access-btnlz\") pod \"ovnkube-node-p7npv\" (UID: \"f4918fd2-0cae-42eb-b8dc-855565f0298b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.500285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.446513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqs5q\" (UniqueName: \"kubernetes.io/projected/0265afbf-1a11-438e-b093-11f6dcfbd075-kube-api-access-cqs5q\") pod \"iptables-alerter-75tj7\" (UID: \"0265afbf-1a11-438e-b093-11f6dcfbd075\") " pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.516054    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7gzr4"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.525634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mm9zg"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:18.531361    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb158c6c_ad86_4462_a1e0_fe975d0055c4.slice/crio-ec17d50ed1bbf2c83aff81f1e0c441d0db30c831ec07e913a919dfd882dedde9 WatchSource:0}: Error finding container ec17d50ed1bbf2c83aff81f1e0c441d0db30c831ec07e913a919dfd882dedde9: Status 404 returned error can't find the container with id ec17d50ed1bbf2c83aff81f1e0c441d0db30c831ec07e913a919dfd882dedde9
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:18.536656    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8220436b_a8e2_4f0e_ab53_776cc9660d6a.slice/crio-18f880ae169565622ca9bfdfc371864ba2261c242adc7da0ab271ffce7da4b43 WatchSource:0}: Error finding container 18f880ae169565622ca9bfdfc371864ba2261c242adc7da0ab271ffce7da4b43: Status 404 returned error can't find the container with id 18f880ae169565622ca9bfdfc371864ba2261c242adc7da0ab271ffce7da4b43
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.543504    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:18.554129    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f19aa3_2b80_4581_b937_4abbb0162b51.slice/crio-7de072dcfe1b99054a79a3ab7cebc75bb8610a9883657d98acaf8dd697d7e599 WatchSource:0}: Error finding container 7de072dcfe1b99054a79a3ab7cebc75bb8610a9883657d98acaf8dd697d7e599: Status 404 returned error can't find the container with id 7de072dcfe1b99054a79a3ab7cebc75bb8610a9883657d98acaf8dd697d7e599
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.554234    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rvc7q"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.560289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5xkmx"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:18.574153    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4516b8_9d8d_48ab_b802_794ad1ed6874.slice/crio-1bff7ca0e768f1de0eb99a59254c7de24dc8beec8f18f7749a888db609608c69 WatchSource:0}: Error finding container 1bff7ca0e768f1de0eb99a59254c7de24dc8beec8f18f7749a888db609608c69: Status 404 returned error can't find the container with id 1bff7ca0e768f1de0eb99a59254c7de24dc8beec8f18f7749a888db609608c69
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:18.578125    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afa7d8f_5b83_494b_9166_c8fd3fb1282b.slice/crio-27080e17e21120311a025583c6696656ff43ba9e43e5f0b030f8c537cb46a774 WatchSource:0}: Error finding container 27080e17e21120311a025583c6696656ff43ba9e43e5f0b030f8c537cb46a774: Status 404 returned error can't find the container with id 27080e17e21120311a025583c6696656ff43ba9e43e5f0b030f8c537cb46a774
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.589633    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75tj7"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.599395    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:18.606509    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0265afbf_1a11_438e_b093_11f6dcfbd075.slice/crio-df0bdb6c0a6e6761a23cfba119e7096da4a578de8730c2c5365020afd2dd24d7 WatchSource:0}: Error finding container df0bdb6c0a6e6761a23cfba119e7096da4a578de8730c2c5365020afd2dd24d7: Status 404 returned error can't find the container with id df0bdb6c0a6e6761a23cfba119e7096da4a578de8730c2c5365020afd2dd24d7
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.611326    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:18.612350    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4918fd2_0cae_42eb_b8dc_855565f0298b.slice/crio-56ae17d90b918de42f9766f74ea51be8f43eba854459f189531af16d5333b2ae WatchSource:0}: Error finding container 56ae17d90b918de42f9766f74ea51be8f43eba854459f189531af16d5333b2ae: Status 404 returned error can't find the container with id 56ae17d90b918de42f9766f74ea51be8f43eba854459f189531af16d5333b2ae
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.617623    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f6mv8"
Jul 09 18:04:18.623108 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:18.621811    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98744d7_a4b6_4f0e_830b_da6c2ba8bbb9.slice/crio-88e69b00ce8c70d28ee4532df2523d60ed31370198afefaa1400cb5d3ab73058 WatchSource:0}: Error finding container 88e69b00ce8c70d28ee4532df2523d60ed31370198afefaa1400cb5d3ab73058: Status 404 returned error can't find the container with id 88e69b00ce8c70d28ee4532df2523d60ed31370198afefaa1400cb5d3ab73058
Jul 09 18:04:18.632993 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:04:18.627622    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33e40f4_ff09_4aa9_9fb1_7c8de50e683a.slice/crio-28f557bce881b388e35adcd62bfcfcc40e94c732e010d4e63e267c5947c5c00d WatchSource:0}: Error finding container 28f557bce881b388e35adcd62bfcfcc40e94c732e010d4e63e267c5947c5c00d: Status 404 returned error can't find the container with id 28f557bce881b388e35adcd62bfcfcc40e94c732e010d4e63e267c5947c5c00d
Jul 09 18:04:18.946175 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.826705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:18.946175 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.826826    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:18.946175 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.826899    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs podName:1803cc34-6cae-4580-a034-b67c1cab8b8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:19.826880951 +0000 UTC m=+3.978518203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs") pod "network-metrics-daemon-zsbvk" (UID: "1803cc34-6cae-4580-a034-b67c1cab8b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:18.946175 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:18.927965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:18.946175 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.928105    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:18.946175 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:18.928167    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret podName:c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:19.9281491 +0000 UTC m=+4.079786356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret") pod "global-pull-secret-syncer-5wc7j" (UID: "c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:19.028894 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.028859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:19.029097 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:19.029073    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:19.029159 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:19.029111    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:19.029159 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:19.029126    2560 projected.go:196] Error preparing data for projected volume kube-api-access-xknv5 for pod openshift-network-diagnostics/network-check-target-gbg4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:19.029238 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:19.029207    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5 podName:882b7841-7d01-46cb-afca-f21a77389624 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:20.029187286 +0000 UTC m=+4.180824538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xknv5" (UniqueName: "kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5") pod "network-check-target-gbg4t" (UID: "882b7841-7d01-46cb-afca-f21a77389624") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:19.271611 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.271497    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:18 +0000 UTC" deadline="2028-02-15 22:05:22.193179842 +0000 UTC"
Jul 09 18:04:19.271611 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.271559    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14068h1m2.921626795s"
Jul 09 18:04:19.398011 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.397934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5xkmx" event={"ID":"9afa7d8f-5b83-494b-9166-c8fd3fb1282b","Type":"ContainerStarted","Data":"27080e17e21120311a025583c6696656ff43ba9e43e5f0b030f8c537cb46a774"}
Jul 09 18:04:19.406307 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.403856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rvc7q" event={"ID":"1b4516b8-9d8d-48ab-b802-794ad1ed6874","Type":"ContainerStarted","Data":"1bff7ca0e768f1de0eb99a59254c7de24dc8beec8f18f7749a888db609608c69"}
Jul 09 18:04:19.407884 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.407856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp" event={"ID":"22f19aa3-2b80-4581-b937-4abbb0162b51","Type":"ContainerStarted","Data":"7de072dcfe1b99054a79a3ab7cebc75bb8610a9883657d98acaf8dd697d7e599"}
Jul 09 18:04:19.409439 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.409412    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75tj7" event={"ID":"0265afbf-1a11-438e-b093-11f6dcfbd075","Type":"ContainerStarted","Data":"df0bdb6c0a6e6761a23cfba119e7096da4a578de8730c2c5365020afd2dd24d7"}
Jul 09 18:04:19.411068 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.411041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mm9zg" event={"ID":"8220436b-a8e2-4f0e-ab53-776cc9660d6a","Type":"ContainerStarted","Data":"18f880ae169565622ca9bfdfc371864ba2261c242adc7da0ab271ffce7da4b43"}
Jul 09 18:04:19.414267 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.414209    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" event={"ID":"bb158c6c-ad86-4462-a1e0-fe975d0055c4","Type":"ContainerStarted","Data":"ec17d50ed1bbf2c83aff81f1e0c441d0db30c831ec07e913a919dfd882dedde9"}
Jul 09 18:04:19.416735 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.416672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f6mv8" event={"ID":"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a","Type":"ContainerStarted","Data":"28f557bce881b388e35adcd62bfcfcc40e94c732e010d4e63e267c5947c5c00d"}
Jul 09 18:04:19.419571 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.419493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8hb8n" event={"ID":"b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9","Type":"ContainerStarted","Data":"88e69b00ce8c70d28ee4532df2523d60ed31370198afefaa1400cb5d3ab73058"}
Jul 09 18:04:19.423547 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.423514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerStarted","Data":"56ae17d90b918de42f9766f74ea51be8f43eba854459f189531af16d5333b2ae"}
Jul 09 18:04:19.836102 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.836062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:19.836343 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:19.836254    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:19.836445 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:19.836356    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs podName:1803cc34-6cae-4580-a034-b67c1cab8b8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.83633246 +0000 UTC m=+5.987969714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs") pod "network-metrics-daemon-zsbvk" (UID: "1803cc34-6cae-4580-a034-b67c1cab8b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:19.937548 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:19.937345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:19.938307 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:19.937856    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:19.938307 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:19.937938    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret podName:c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:21.937916222 +0000 UTC m=+6.089553486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret") pod "global-pull-secret-syncer-5wc7j" (UID: "c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:20.040740 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:20.040701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:20.040990 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:20.040939    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:20.040990 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:20.040965    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:20.040990 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:20.040978    2560 projected.go:196] Error preparing data for projected volume kube-api-access-xknv5 for pod openshift-network-diagnostics/network-check-target-gbg4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:20.041190 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:20.041036    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5 podName:882b7841-7d01-46cb-afca-f21a77389624 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:22.041016756 +0000 UTC m=+6.192654012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xknv5" (UniqueName: "kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5") pod "network-check-target-gbg4t" (UID: "882b7841-7d01-46cb-afca-f21a77389624") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:20.272659 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:20.272536    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:18 +0000 UTC" deadline="2028-03-18 02:00:18.403951966 +0000 UTC"
Jul 09 18:04:20.272659 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:20.272608    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14815h55m58.131347754s"
Jul 09 18:04:20.383735 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:20.383700    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:20.383908 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:20.383843    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:20.384308 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:20.384284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:20.384419 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:20.384400    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:20.384701 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:20.384682    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:20.384799 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:20.384778    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:21.859980 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:21.859943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:21.860481 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:21.860167    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:21.860481 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:21.860331    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs podName:1803cc34-6cae-4580-a034-b67c1cab8b8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:25.860214174 +0000 UTC m=+10.011851430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs") pod "network-metrics-daemon-zsbvk" (UID: "1803cc34-6cae-4580-a034-b67c1cab8b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:21.960813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:21.960779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:21.960987 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:21.960939    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:21.961091 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:21.961000    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret podName:c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:25.960980153 +0000 UTC m=+10.112617405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret") pod "global-pull-secret-syncer-5wc7j" (UID: "c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:22.061723 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:22.061665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:22.061884 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:22.061857    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:22.061884 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:22.061880    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:22.062018 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:22.061893    2560 projected.go:196] Error preparing data for projected volume kube-api-access-xknv5 for pod openshift-network-diagnostics/network-check-target-gbg4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:22.062018 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:22.061948    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5 podName:882b7841-7d01-46cb-afca-f21a77389624 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.061930169 +0000 UTC m=+10.213567423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xknv5" (UniqueName: "kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5") pod "network-check-target-gbg4t" (UID: "882b7841-7d01-46cb-afca-f21a77389624") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:22.386871 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:22.385915    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:22.386871 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:22.386053    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:22.386871 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:22.386442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:22.386871 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:22.386583    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:22.386871 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:22.386636    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:22.386871 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:22.386707    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:24.383184 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:24.383111    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:24.383528 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:24.383250    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:24.383755 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:24.383728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:24.383888 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:24.383856    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:24.383963 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:24.383943    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:24.384103 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:24.384075    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:25.896022 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:25.895988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:25.896517 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:25.896178    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:25.896517 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:25.896273    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs podName:1803cc34-6cae-4580-a034-b67c1cab8b8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:33.896252767 +0000 UTC m=+18.047890043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs") pod "network-metrics-daemon-zsbvk" (UID: "1803cc34-6cae-4580-a034-b67c1cab8b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:25.996846 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:25.996805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:25.997010 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:25.996994    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:25.997074 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:25.997055    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret podName:c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:33.997037634 +0000 UTC m=+18.148674889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret") pod "global-pull-secret-syncer-5wc7j" (UID: "c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:26.097547 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:26.097510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:26.097729 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:26.097691    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:26.097729 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:26.097718    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:26.097829 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:26.097732    2560 projected.go:196] Error preparing data for projected volume kube-api-access-xknv5 for pod openshift-network-diagnostics/network-check-target-gbg4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:26.097829 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:26.097785    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5 podName:882b7841-7d01-46cb-afca-f21a77389624 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:34.097766399 +0000 UTC m=+18.249403652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xknv5" (UniqueName: "kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5") pod "network-check-target-gbg4t" (UID: "882b7841-7d01-46cb-afca-f21a77389624") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:26.384475 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:26.384398    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:26.384646 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:26.384525    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:26.384646 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:26.384547    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:26.384751 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:26.384652    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:26.384751 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:26.384709    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:26.384845 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:26.384791    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:28.383858 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:28.382935    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:28.383858 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:28.383275    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:28.383858 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:28.383695    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:28.383858 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:28.383808    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:28.384781 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:28.384352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:28.384781 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:28.384449    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:30.382860 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:30.382812    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:30.382860 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:30.382842    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:30.383357 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:30.382812    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:30.383357 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:30.382953    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:30.383357 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:30.383050    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:30.383357 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:30.383130    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:32.383076 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:32.383041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:32.383525 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:32.383091    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:32.383525 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:32.383041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:32.383525 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:32.383184    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:32.383525 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:32.383275    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:32.383525 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:32.383378    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:33.957955 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:33.957922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:33.958366 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:33.958033    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:33.958366 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:33.958094    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs podName:1803cc34-6cae-4580-a034-b67c1cab8b8e nodeName:}" failed. No retries permitted until 2026-07-09 18:04:49.958078054 +0000 UTC m=+34.109715322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs") pod "network-metrics-daemon-zsbvk" (UID: "1803cc34-6cae-4580-a034-b67c1cab8b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:34.058247 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:34.058219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:34.058385 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:34.058341    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:34.058422 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:34.058394    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret podName:c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:50.058379159 +0000 UTC m=+34.210016429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret") pod "global-pull-secret-syncer-5wc7j" (UID: "c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:34.159151 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:34.159117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:34.159289 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:34.159230    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:34.159289 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:34.159243    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:34.159289 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:34.159251    2560 projected.go:196] Error preparing data for projected volume kube-api-access-xknv5 for pod openshift-network-diagnostics/network-check-target-gbg4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:34.159411 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:34.159293    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5 podName:882b7841-7d01-46cb-afca-f21a77389624 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:50.159282038 +0000 UTC m=+34.310919291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xknv5" (UniqueName: "kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5") pod "network-check-target-gbg4t" (UID: "882b7841-7d01-46cb-afca-f21a77389624") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:34.383568 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:34.383529    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:34.383568 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:34.383570    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:34.383815 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:34.383529    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:34.383815 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:34.383664    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:34.383815 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:34.383765    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:34.383972 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:34.383895    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:36.384079 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:36.384049    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:36.384530 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:36.384169    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:36.384530 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:36.384215    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:36.384530 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:36.384273    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:36.384530 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:36.384304    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:36.384530 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:36.384370    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:38.384830 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:38.384285    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:38.384830 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:38.384418    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:38.384830 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:38.384464    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:38.384830 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:38.384593    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:38.384830 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:38.384664    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:38.384830 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:38.384790    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:38.472295 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:38.472220    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal" event={"ID":"6f2dcd2c4647c8a5c8152fee259ad6a9","Type":"ContainerStarted","Data":"9985a6289047f9dded1f030f032f268ed1428f3e2f007fe7548364e88aa3b39a"}
Jul 09 18:04:38.474116 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:38.474086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rvc7q" event={"ID":"1b4516b8-9d8d-48ab-b802-794ad1ed6874","Type":"ContainerStarted","Data":"3dc61c718b4f9f4a4db7fe5ec0f8447f1b552805df840e97efd0388eb870fcad"}
Jul 09 18:04:38.488945 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:38.488894    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-87.ec2.internal" podStartSLOduration=21.488856781 podStartE2EDuration="21.488856781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:38.48879606 +0000 UTC m=+22.640433333" watchObservedRunningTime="2026-07-09 18:04:38.488856781 +0000 UTC m=+22.640494053"
Jul 09 18:04:39.478680 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.478240    2560 generic.go:356] "Generic (PLEG): container finished" podID="bb158c6c-ad86-4462-a1e0-fe975d0055c4" containerID="5ec9d86b81b39ac63176a3f0b5dafd7d1056761b2c0c3cad8492a755d5d8fd02" exitCode=0
Jul 09 18:04:39.479436 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.478320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" event={"ID":"bb158c6c-ad86-4462-a1e0-fe975d0055c4","Type":"ContainerDied","Data":"5ec9d86b81b39ac63176a3f0b5dafd7d1056761b2c0c3cad8492a755d5d8fd02"}
Jul 09 18:04:39.480399 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.480375    2560 generic.go:356] "Generic (PLEG): container finished" podID="291def8d0af3fe3d1b87f627e9c4338e" containerID="bcb845f0c536ac7d35b0fd77e83373ffca1b5086540d2d0ce8b02d362d185fa2" exitCode=0
Jul 09 18:04:39.480470 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.480455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal" event={"ID":"291def8d0af3fe3d1b87f627e9c4338e","Type":"ContainerDied","Data":"bcb845f0c536ac7d35b0fd77e83373ffca1b5086540d2d0ce8b02d362d185fa2"}
Jul 09 18:04:39.481927 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.481902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f6mv8" event={"ID":"c33e40f4-ff09-4aa9-9fb1-7c8de50e683a","Type":"ContainerStarted","Data":"84c8ddc72113d52fa46c1eeda799fa206287d3a307ea20a9a23191aaa119aec3"}
Jul 09 18:04:39.483255 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.483238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8hb8n" event={"ID":"b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9","Type":"ContainerStarted","Data":"1c64d7a7a5063de6a27ebf824945f702dd3923ab0250a16712abec83b7186730"}
Jul 09 18:04:39.487406 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.487388    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:04:39.487883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.487866    2560 generic.go:356] "Generic (PLEG): container finished" podID="f4918fd2-0cae-42eb-b8dc-855565f0298b" containerID="7e0a7ca6a4c816a97c76e9cbcbf53d2c85d7150d82a302f5e430ee52689dc69a" exitCode=1
Jul 09 18:04:39.487970 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.487927    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerStarted","Data":"4724d96b2d65d1b699c0283e8da6457d7de6415f21907daf87183aba82ff4774"}
Jul 09 18:04:39.488027 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.488007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerStarted","Data":"6ee6a6945e77a3b98033e65c6c2c8f06a0c19daf2c795f6916219612d062b80e"}
Jul 09 18:04:39.488077 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.488060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerStarted","Data":"26b8e0ef3a5f9189a2d8c5fcd2e098d977a07ae73437da0882c302307c1a3bad"}
Jul 09 18:04:39.488128 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.488108    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerStarted","Data":"61a72612b650b12d7f9086a763d464dfa94e66ea90926f20b6dfd5d21afc18f1"}
Jul 09 18:04:39.488177 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.488158    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerDied","Data":"7e0a7ca6a4c816a97c76e9cbcbf53d2c85d7150d82a302f5e430ee52689dc69a"}
Jul 09 18:04:39.488231 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.488207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerStarted","Data":"2bb4e41235200dbc2b1be2670aba1e58a44c8ee453364433bfc72ae2fa67b1e2"}
Jul 09 18:04:39.489005 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.488988    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5xkmx" event={"ID":"9afa7d8f-5b83-494b-9166-c8fd3fb1282b","Type":"ContainerStarted","Data":"e6b4ffb198a4000d1d0ebd8970a6f556ab6f1d46fc647fb263ddcad2aee95b34"}
Jul 09 18:04:39.490310 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.490292    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp" event={"ID":"22f19aa3-2b80-4581-b937-4abbb0162b51","Type":"ContainerStarted","Data":"da89a2858e6e0b5c9dc1a0950461fb22f940ae17b81ec449812cfcb8d43b2060"}
Jul 09 18:04:39.491454 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.491434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mm9zg" event={"ID":"8220436b-a8e2-4f0e-ab53-776cc9660d6a","Type":"ContainerStarted","Data":"433a5683b71b33bee3aa430768ee2354c7c107ba2dd4edc41d39fb8c37958c51"}
Jul 09 18:04:39.504087 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.504040    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rvc7q" podStartSLOduration=2.743793509 podStartE2EDuration="22.504024616s" totalImagesPullingTime="19.760231107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:38.510576198 +0000 UTC m=+22.662213471" watchObservedRunningTime="2026-07-09 18:04:39.504024616 +0000 UTC m=+23.655661889"
Jul 09 18:04:39.522673 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.522625    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f6mv8" podStartSLOduration=2.8158022259999997 podStartE2EDuration="22.522610028s" totalImagesPullingTime="19.706807802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:39.522277651 +0000 UTC m=+23.673914925" watchObservedRunningTime="2026-07-09 18:04:39.522610028 +0000 UTC m=+23.674247297"
Jul 09 18:04:39.559430 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.559381    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8hb8n" podStartSLOduration=7.163560689 podStartE2EDuration="22.559359673s" totalImagesPullingTime="15.395798984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:39.558940455 +0000 UTC m=+23.710577752" watchObservedRunningTime="2026-07-09 18:04:39.559359673 +0000 UTC m=+23.710996950"
Jul 09 18:04:39.580352 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.580295    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mm9zg" podStartSLOduration=3.728474136 podStartE2EDuration="23.580274288s" totalImagesPullingTime="19.851800152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:39.580093224 +0000 UTC m=+23.731730517" watchObservedRunningTime="2026-07-09 18:04:39.580274288 +0000 UTC m=+23.731911561"
Jul 09 18:04:39.597790 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.597744    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5xkmx" podStartSLOduration=2.8414233429999998 podStartE2EDuration="22.597728969s" totalImagesPullingTime="19.756305626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:39.597571455 +0000 UTC m=+23.749208756" watchObservedRunningTime="2026-07-09 18:04:39.597728969 +0000 UTC m=+23.749366240"
Jul 09 18:04:39.745986 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.745945    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:39.817520 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.817469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:39.972600 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:39.972567    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:04:40.281665 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.281566    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:04:39.972597202Z","UUID":"cf149fa6-8dd4-4c9e-9ea8-85ca6886ab48","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:04:40.283120 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.283100    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:04:40.283232 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.283135    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:04:40.382857 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.382824    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:40.383044 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.382824    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:40.383044 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:40.382950    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:40.383158 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:40.383043    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:40.383158 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.382833    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:40.383158 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:40.383144    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:40.494862 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.494824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp" event={"ID":"22f19aa3-2b80-4581-b937-4abbb0162b51","Type":"ContainerStarted","Data":"249a704afd225d354d295ed62084f58ae887195b3d328cb10e02d4ff3c8340fe"}
Jul 09 18:04:40.496216 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.496193    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75tj7" event={"ID":"0265afbf-1a11-438e-b093-11f6dcfbd075","Type":"ContainerStarted","Data":"e44e51c9941b1f6d8b806a7e3e9d9a2a465994f97a9fdd3dc2319c4649e34e5d"}
Jul 09 18:04:40.498005 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.497976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal" event={"ID":"291def8d0af3fe3d1b87f627e9c4338e","Type":"ContainerStarted","Data":"c896fa8d9bdbd6cda18d93a83b9dd308cbca6745567be0a4915b552b959831a4"}
Jul 09 18:04:40.608473 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.608368    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-75tj7" podStartSLOduration=3.8814115620000003 podStartE2EDuration="23.60835128s" totalImagesPullingTime="19.726939718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:40.558780282 +0000 UTC m=+24.710417555" watchObservedRunningTime="2026-07-09 18:04:40.60835128 +0000 UTC m=+24.759988553"
Jul 09 18:04:40.619319 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.619282    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:40.700233 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.700193    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:40.891246 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.891145    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:40.992555 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:40.992517    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:41.368162 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:41.367976    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:41.471420 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:41.471389    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:41.503284 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:41.503246    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:04:41.503710 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:41.503686    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerStarted","Data":"6ea3ca7b864a9f9249ee48df180c724dc13020502fbaca6af7b28c2c593bdb19"}
Jul 09 18:04:41.505560 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:41.505533    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp" event={"ID":"22f19aa3-2b80-4581-b937-4abbb0162b51","Type":"ContainerStarted","Data":"a3b394d8313441203fa787445748ee35087e48879e967b821f90a3f183b05d2c"}
Jul 09 18:04:41.558283 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:41.558228    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-87.ec2.internal" podStartSLOduration=24.558208105 podStartE2EDuration="24.558208105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:40.611878127 +0000 UTC m=+24.763515399" watchObservedRunningTime="2026-07-09 18:04:41.558208105 +0000 UTC m=+25.709845378"
Jul 09 18:04:41.558525 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:41.558498    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qr5bp" podStartSLOduration=2.309218573 podStartE2EDuration="24.558472516s" totalImagesPullingTime="22.249253943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:41.557062819 +0000 UTC m=+25.708700095" watchObservedRunningTime="2026-07-09 18:04:41.558472516 +0000 UTC m=+25.710109791"
Jul 09 18:04:41.592797 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:41.592771    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5xkmx_9afa7d8f-5b83-494b-9166-c8fd3fb1282b/dns-node-resolver/0.log"
Jul 09 18:04:42.383762 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:42.383722    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:42.383762 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:42.383751    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:42.384057 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:42.383759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:42.384057 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:42.383874    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:42.384057 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:42.384025    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:42.384174 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:42.384121    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:42.589406 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:42.589381    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f6mv8_c33e40f4-ff09-4aa9-9fb1-7c8de50e683a/node-ca/0.log"
Jul 09 18:04:43.182913 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:43.182879    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:43.513609 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:43.513441    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:04:43.513962 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:43.513938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerStarted","Data":"3e804cc6c925e0437e0b918bfa86a3d6367cb551964e5b2b2811a7c5cc0b1d8c"}
Jul 09 18:04:43.514241 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:43.514221    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:43.514384 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:43.514293    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:43.514503 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:43.514471    2560 scope.go:117] "RemoveContainer" containerID="7e0a7ca6a4c816a97c76e9cbcbf53d2c85d7150d82a302f5e430ee52689dc69a"
Jul 09 18:04:43.532009 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:43.531974    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:43.794970 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:43.794885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:43.926114 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:43.926073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:44.002463 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:44.002430    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:44.003216 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:44.003200    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:44.030799 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:44.030759    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:44.377951 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:44.377862    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:44.378528 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:44.378501    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8hb8n"
Jul 09 18:04:44.382764 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:44.382666    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:44.382764 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:44.382689    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:44.382764 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:44.382672    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:44.383006 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:44.382787    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:44.383006 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:44.382869    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:44.383006 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:44.382968    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:44.745508 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:44.745457    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:45.084895 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.084824    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:45.277245 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.277209    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:45.405139 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.405050    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:45.522131 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.522108    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:04:45.522467 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.522441    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" event={"ID":"f4918fd2-0cae-42eb-b8dc-855565f0298b","Type":"ContainerStarted","Data":"8eda4ecf481280a14c3ff4cf4d8a8c658ef56fb25fea62e7729adc6f6c74b9ef"}
Jul 09 18:04:45.522710 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.522694    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:45.536963 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.536937    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:04:45.595743 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.595716    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:45.671052 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.670952    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv" podStartSLOduration=8.887208576 podStartE2EDuration="28.670931668s" totalImagesPullingTime="19.783723092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:45.622111907 +0000 UTC m=+29.773749172" watchObservedRunningTime="2026-07-09 18:04:45.670931668 +0000 UTC m=+29.822568939"
Jul 09 18:04:45.817835 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.817797    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:45.925975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:45.925906    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:46.129620 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:46.129428    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbg4t"]
Jul 09 18:04:46.130007 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:46.129696    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:46.130007 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:46.129788    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:46.142190 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:46.142154    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5wc7j"]
Jul 09 18:04:46.142356 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:46.142288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:46.142422 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:46.142399    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:46.143078 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:46.143053    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zsbvk"]
Jul 09 18:04:46.143189 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:46.143177    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:46.143274 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:46.143258    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:46.526244 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:46.526191    2560 generic.go:356] "Generic (PLEG): container finished" podID="bb158c6c-ad86-4462-a1e0-fe975d0055c4" containerID="ae83514daaa479908ce1211520c19b97817bc88bf5e28546b0d236c5d665be8b" exitCode=0
Jul 09 18:04:46.526461 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:46.526273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" event={"ID":"bb158c6c-ad86-4462-a1e0-fe975d0055c4","Type":"ContainerDied","Data":"ae83514daaa479908ce1211520c19b97817bc88bf5e28546b0d236c5d665be8b"}
Jul 09 18:04:46.768937 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:46.768902    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:47.188923 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:47.188891    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:47.299425 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:47.299396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:47.383389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:47.383364    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:47.383543 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:47.383500    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:47.394753 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:47.394723    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:47.574753 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:47.574721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:47.780616 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:47.780448    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:48.346538 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:48.346499    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:48.383604 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:48.383509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:48.383604 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:48.383543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:48.383812 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:48.383629    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:48.383812 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:48.383748    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:48.464859 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:48.464828    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:48.532640 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:48.532606    2560 generic.go:356] "Generic (PLEG): container finished" podID="bb158c6c-ad86-4462-a1e0-fe975d0055c4" containerID="526cb9a3339c188d83b8f8991eb67326c7a6cd90e13aefbb03f47845dc9095dd" exitCode=0
Jul 09 18:04:48.532640 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:48.532644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" event={"ID":"bb158c6c-ad86-4462-a1e0-fe975d0055c4","Type":"ContainerDied","Data":"526cb9a3339c188d83b8f8991eb67326c7a6cd90e13aefbb03f47845dc9095dd"}
Jul 09 18:04:48.954467 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:48.954433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:49.063033 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:49.062996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:49.163876 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:49.163833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:49.375207 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:49.375175    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:49.383199 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:49.383174    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:49.383302 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:49.383283    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:49.667912 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:49.667833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:49.765900 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:49.765865    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:49.866639 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:49.866600    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:49.981585 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:49.981510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:49.981713 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:49.981618    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:49.981713 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:49.981669    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs podName:1803cc34-6cae-4580-a034-b67c1cab8b8e nodeName:}" failed. No retries permitted until 2026-07-09 18:05:21.981654471 +0000 UTC m=+66.133291721 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs") pod "network-metrics-daemon-zsbvk" (UID: "1803cc34-6cae-4580-a034-b67c1cab8b8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:50.082240 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:50.082207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:50.082370 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:50.082353    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:50.082423 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:50.082412    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret podName:c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:22.082398819 +0000 UTC m=+66.234036068 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret") pod "global-pull-secret-syncer-5wc7j" (UID: "c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:50.183182 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:50.183152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:50.183321 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:50.183280    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:50.183321 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:50.183295    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:50.183321 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:50.183302    2560 projected.go:196] Error preparing data for projected volume kube-api-access-xknv5 for pod openshift-network-diagnostics/network-check-target-gbg4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:50.183453 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:50.183346    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5 podName:882b7841-7d01-46cb-afca-f21a77389624 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:22.183333577 +0000 UTC m=+66.334970831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xknv5" (UniqueName: "kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5") pod "network-check-target-gbg4t" (UID: "882b7841-7d01-46cb-afca-f21a77389624") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:50.382659 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:50.382625    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:50.383089 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:50.382625    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:50.383089 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:50.382753    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:50.383089 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:50.382861    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:50.539810 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:50.539772    2560 generic.go:356] "Generic (PLEG): container finished" podID="bb158c6c-ad86-4462-a1e0-fe975d0055c4" containerID="b51de751d54c87236775962e05e03aa50007b2f58db38fc41abcb6e7a5281e31" exitCode=0
Jul 09 18:04:50.539952 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:50.539857    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" event={"ID":"bb158c6c-ad86-4462-a1e0-fe975d0055c4","Type":"ContainerDied","Data":"b51de751d54c87236775962e05e03aa50007b2f58db38fc41abcb6e7a5281e31"}
Jul 09 18:04:50.571668 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:50.571638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:51.372762 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:51.372716    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:51.383424 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:51.383399    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:51.383989 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:51.383531    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:51.474045 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:51.474007    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:51.579041 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:51.578986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:51.773193 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:51.773090    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:52.165337 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:52.165248    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:52.382732 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:52.382694    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:52.382921 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:52.382831    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:52.382921 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:52.382889    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:52.383033 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:52.382994    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:53.176004 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:53.175959    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:53.382959 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:53.382926    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:53.383139 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:53.383064    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:53.598597 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:53.598561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:53.768462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:53.768423    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:53.834150 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:53.834121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:53.981681 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:53.981575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:54.381407 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:54.381367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:54.383216 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:54.383192    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:54.383321 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:54.383282    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:54.383383 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:54.383369    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:54.383508 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:54.383473    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:54.551650 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:54.551603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" event={"ID":"bb158c6c-ad86-4462-a1e0-fe975d0055c4","Type":"ContainerStarted","Data":"2709f8ab7ac1b2e4ad272f8ab5605d1f059b8032ab08d13e6141592f54c36fd8"}
Jul 09 18:04:55.168913 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:55.168878    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:55.383225 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:55.383192    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:55.383623 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:55.383296    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:55.556758 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:55.556724    2560 generic.go:356] "Generic (PLEG): container finished" podID="bb158c6c-ad86-4462-a1e0-fe975d0055c4" containerID="2709f8ab7ac1b2e4ad272f8ab5605d1f059b8032ab08d13e6141592f54c36fd8" exitCode=0
Jul 09 18:04:55.556922 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:55.556771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" event={"ID":"bb158c6c-ad86-4462-a1e0-fe975d0055c4","Type":"ContainerDied","Data":"2709f8ab7ac1b2e4ad272f8ab5605d1f059b8032ab08d13e6141592f54c36fd8"}
Jul 09 18:04:55.879058 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:55.878800    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:55.985129 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:55.985097    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:56.142292 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:56.142192    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:56.258973 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:56.258938    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:56.383997 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:56.383964    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:56.384367 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:56.384067    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:56.384367 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:56.384158    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:56.384367 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:56.384235    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:56.561414 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:56.561378    2560 generic.go:356] "Generic (PLEG): container finished" podID="bb158c6c-ad86-4462-a1e0-fe975d0055c4" containerID="553c1cb5101eed0e60731e219621ce1e19062426702636ddfe9260b91ec8e619" exitCode=0
Jul 09 18:04:56.561611 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:56.561417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" event={"ID":"bb158c6c-ad86-4462-a1e0-fe975d0055c4","Type":"ContainerDied","Data":"553c1cb5101eed0e60731e219621ce1e19062426702636ddfe9260b91ec8e619"}
Jul 09 18:04:56.779134 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:56.779089    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:56.852635 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:56.852552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:56.976274 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:56.976238    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:57.383551 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:57.383516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:57.383724 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:57.383620    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:04:57.405109 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:57.405079    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:57.567256 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:57.567225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" event={"ID":"bb158c6c-ad86-4462-a1e0-fe975d0055c4","Type":"ContainerStarted","Data":"75dfdde205482484ef788c80358c443d301a4fe033c812e72cfdc560be434d06"}
Jul 09 18:04:57.593890 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:57.593840    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7gzr4" podStartSLOduration=9.395738691 podStartE2EDuration="41.593825025s" totalImagesPullingTime="32.198086334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:57.593344944 +0000 UTC m=+41.744982215" watchObservedRunningTime="2026-07-09 18:04:57.593825025 +0000 UTC m=+41.745462297"
Jul 09 18:04:58.383617 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:58.383580    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:04:58.383617 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:58.383609    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:04:58.383819 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:58.383691    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:04:58.383819 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:58.383756    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:04:59.382901 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:04:59.382864    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:04:59.383257 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:04:59.382975    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:05:00.383601 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:00.383557    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:05:00.383601 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:00.383591    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:05:00.384100 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:00.383698    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:05:00.384100 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:00.383791    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:05:01.383537 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:01.383500    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:05:01.383738 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:01.383637    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:05:02.264060 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:02.264028    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:02.350885 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:02.350853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:02.383443 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:02.383411    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:05:02.383612 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:02.383563    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:05:02.383653 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:02.383614    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:05:02.383744 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:02.383724    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:05:03.383059 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:03.383029    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:05:03.383225 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:03.383125    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsbvk" podUID="1803cc34-6cae-4580-a034-b67c1cab8b8e"
Jul 09 18:05:04.383180 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:04.383141    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:05:04.383671 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:04.383278    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbg4t" podUID="882b7841-7d01-46cb-afca-f21a77389624"
Jul 09 18:05:04.383671 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:04.383321    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:05:04.383671 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:04.383405    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wc7j" podUID="c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8"
Jul 09 18:05:04.729693 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:04.729614    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:04.925638 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:04.925603    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:05.039031 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.038994    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:05.146096 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.146056    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:05.206929 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.206905    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-87.ec2.internal" event="NodeReady"
Jul 09 18:05:05.207111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.207049    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:05:05.246914 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.246881    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:05.265532 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.265469    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b74d8bc8-dg69n"]
Jul 09 18:05:05.291164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.291091    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.299090 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.299070    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:05:05.299360 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.299343    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:05:05.309339 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.309313    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t5pfs\""
Jul 09 18:05:05.309444 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.309319    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:05:05.343240 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.343215    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b74d8bc8-dg69n"]
Jul 09 18:05:05.344196 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.344174    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:05:05.359664 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.359638    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zrxtr"]
Jul 09 18:05:05.380455 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.380429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.382661 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.382639    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:05:05.392328 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392306    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:05:05.392779 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392723    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmsf\" (UniqueName: \"kubernetes.io/projected/1e0cc46f-0a9d-4139-adb0-969e5056fe79-kube-api-access-wmmsf\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.392779 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e0cc46f-0a9d-4139-adb0-969e5056fe79-registry-certificates\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.392954 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e0cc46f-0a9d-4139-adb0-969e5056fe79-registry-tls\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.392954 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392736    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zrxtr"]
Jul 09 18:05:05.392954 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pgfb\" (UniqueName: \"kubernetes.io/projected/d40f86e5-325b-4c1f-8468-59b44d58b0c4-kube-api-access-2pgfb\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.392954 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392844    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:05:05.392954 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e0cc46f-0a9d-4139-adb0-969e5056fe79-installation-pull-secrets\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.392954 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392871    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mw97p\""
Jul 09 18:05:05.392954 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40f86e5-325b-4c1f-8468-59b44d58b0c4-tmp-dir\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.393366 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.392983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e0cc46f-0a9d-4139-adb0-969e5056fe79-image-registry-private-configuration\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.393366 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.393024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e0cc46f-0a9d-4139-adb0-969e5056fe79-trusted-ca\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.393366 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.393036    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:05.393366 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.393100    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e0cc46f-0a9d-4139-adb0-969e5056fe79-ca-trust-extracted\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.393366 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.393130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e0cc46f-0a9d-4139-adb0-969e5056fe79-bound-sa-token\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.393366 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.393156    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d40f86e5-325b-4c1f-8468-59b44d58b0c4-config-volume\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.393366 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.393206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d40f86e5-325b-4c1f-8468-59b44d58b0c4-metrics-tls\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.393655 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.393560    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vl9c6\""
Jul 09 18:05:05.397050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.397021    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:05.426565 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.426538    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mhsft"]
Jul 09 18:05:05.448811 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.448774    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jnp7h"]
Jul 09 18:05:05.448942 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.448918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhsft"
Jul 09 18:05:05.451406 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.451367    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:05:05.451581 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.451558    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:05:05.451711 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.451692    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:05:05.451784 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.451710    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2bhtg\""
Jul 09 18:05:05.468459 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.468435    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhsft"]
Jul 09 18:05:05.468572 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.468556    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.472537 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.472512    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-htlzv\""
Jul 09 18:05:05.472681 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.472564    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:05:05.472681 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.472523    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:05:05.472681 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.472636    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:05:05.472823 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.472715    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:05:05.484827 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.484804    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jnp7h"]
Jul 09 18:05:05.494312 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pgfb\" (UniqueName: \"kubernetes.io/projected/d40f86e5-325b-4c1f-8468-59b44d58b0c4-kube-api-access-2pgfb\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.494412 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e494343-ee4c-4a4f-8989-24a3e4bda412-data-volume\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.494412 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e0cc46f-0a9d-4139-adb0-969e5056fe79-installation-pull-secrets\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.494412 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40f86e5-325b-4c1f-8468-59b44d58b0c4-tmp-dir\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.494412 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494407    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dn5h\" (UniqueName: \"kubernetes.io/projected/0f45cd8f-7df7-4798-a79e-b292782eef6f-kube-api-access-4dn5h\") pod \"ingress-canary-mhsft\" (UID: \"0f45cd8f-7df7-4798-a79e-b292782eef6f\") " pod="openshift-ingress-canary/ingress-canary-mhsft"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e494343-ee4c-4a4f-8989-24a3e4bda412-crio-socket\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e0cc46f-0a9d-4139-adb0-969e5056fe79-image-registry-private-configuration\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e0cc46f-0a9d-4139-adb0-969e5056fe79-trusted-ca\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e0cc46f-0a9d-4139-adb0-969e5056fe79-ca-trust-extracted\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e0cc46f-0a9d-4139-adb0-969e5056fe79-bound-sa-token\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d40f86e5-325b-4c1f-8468-59b44d58b0c4-config-volume\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d40f86e5-325b-4c1f-8468-59b44d58b0c4-metrics-tls\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmsf\" (UniqueName: \"kubernetes.io/projected/1e0cc46f-0a9d-4139-adb0-969e5056fe79-kube-api-access-wmmsf\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40f86e5-325b-4c1f-8468-59b44d58b0c4-tmp-dir\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhp62\" (UniqueName: \"kubernetes.io/projected/4e494343-ee4c-4a4f-8989-24a3e4bda412-kube-api-access-nhp62\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494797    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e0cc46f-0a9d-4139-adb0-969e5056fe79-registry-certificates\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e494343-ee4c-4a4f-8989-24a3e4bda412-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e494343-ee4c-4a4f-8989-24a3e4bda412-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e0cc46f-0a9d-4139-adb0-969e5056fe79-registry-tls\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f45cd8f-7df7-4798-a79e-b292782eef6f-cert\") pod \"ingress-canary-mhsft\" (UID: \"0f45cd8f-7df7-4798-a79e-b292782eef6f\") " pod="openshift-ingress-canary/ingress-canary-mhsft"
Jul 09 18:05:05.494975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.494959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e0cc46f-0a9d-4139-adb0-969e5056fe79-ca-trust-extracted\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.495786 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.495209    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d40f86e5-325b-4c1f-8468-59b44d58b0c4-config-volume\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.495786 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.495677    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e0cc46f-0a9d-4139-adb0-969e5056fe79-trusted-ca\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.495786 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.495699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e0cc46f-0a9d-4139-adb0-969e5056fe79-registry-certificates\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.499458 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.499435    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d40f86e5-325b-4c1f-8468-59b44d58b0c4-metrics-tls\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.499561 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.499511    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e0cc46f-0a9d-4139-adb0-969e5056fe79-installation-pull-secrets\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.499561 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.499532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e0cc46f-0a9d-4139-adb0-969e5056fe79-registry-tls\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.499667 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.499647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e0cc46f-0a9d-4139-adb0-969e5056fe79-image-registry-private-configuration\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.527423 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.527398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmsf\" (UniqueName: \"kubernetes.io/projected/1e0cc46f-0a9d-4139-adb0-969e5056fe79-kube-api-access-wmmsf\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.528003 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.527987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pgfb\" (UniqueName: \"kubernetes.io/projected/d40f86e5-325b-4c1f-8468-59b44d58b0c4-kube-api-access-2pgfb\") pod \"dns-default-zrxtr\" (UID: \"d40f86e5-325b-4c1f-8468-59b44d58b0c4\") " pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.539347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.539323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e0cc46f-0a9d-4139-adb0-969e5056fe79-bound-sa-token\") pod \"image-registry-5b74d8bc8-dg69n\" (UID: \"1e0cc46f-0a9d-4139-adb0-969e5056fe79\") " pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.595577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.595496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dn5h\" (UniqueName: \"kubernetes.io/projected/0f45cd8f-7df7-4798-a79e-b292782eef6f-kube-api-access-4dn5h\") pod \"ingress-canary-mhsft\" (UID: \"0f45cd8f-7df7-4798-a79e-b292782eef6f\") " pod="openshift-ingress-canary/ingress-canary-mhsft"
Jul 09 18:05:05.595577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.595532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e494343-ee4c-4a4f-8989-24a3e4bda412-crio-socket\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.595766 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.595606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhp62\" (UniqueName: \"kubernetes.io/projected/4e494343-ee4c-4a4f-8989-24a3e4bda412-kube-api-access-nhp62\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.595766 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.595635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e494343-ee4c-4a4f-8989-24a3e4bda412-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.595766 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.595657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e494343-ee4c-4a4f-8989-24a3e4bda412-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.595766 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.595685    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f45cd8f-7df7-4798-a79e-b292782eef6f-cert\") pod \"ingress-canary-mhsft\" (UID: \"0f45cd8f-7df7-4798-a79e-b292782eef6f\") " pod="openshift-ingress-canary/ingress-canary-mhsft"
Jul 09 18:05:05.595766 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.595748    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e494343-ee4c-4a4f-8989-24a3e4bda412-data-volume\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.595985 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.595769    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e494343-ee4c-4a4f-8989-24a3e4bda412-crio-socket\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.596096 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.596077    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e494343-ee4c-4a4f-8989-24a3e4bda412-data-volume\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.596254 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.596236    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e494343-ee4c-4a4f-8989-24a3e4bda412-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.598012 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.597991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e494343-ee4c-4a4f-8989-24a3e4bda412-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.598163 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.598147    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f45cd8f-7df7-4798-a79e-b292782eef6f-cert\") pod \"ingress-canary-mhsft\" (UID: \"0f45cd8f-7df7-4798-a79e-b292782eef6f\") " pod="openshift-ingress-canary/ingress-canary-mhsft"
Jul 09 18:05:05.600040 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.600026    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:05.616625 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.616597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhp62\" (UniqueName: \"kubernetes.io/projected/4e494343-ee4c-4a4f-8989-24a3e4bda412-kube-api-access-nhp62\") pod \"insights-runtime-extractor-jnp7h\" (UID: \"4e494343-ee4c-4a4f-8989-24a3e4bda412\") " pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.616713 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.616626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dn5h\" (UniqueName: \"kubernetes.io/projected/0f45cd8f-7df7-4798-a79e-b292782eef6f-kube-api-access-4dn5h\") pod \"ingress-canary-mhsft\" (UID: \"0f45cd8f-7df7-4798-a79e-b292782eef6f\") " pod="openshift-ingress-canary/ingress-canary-mhsft"
Jul 09 18:05:05.690546 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.690511    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:05.744977 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.744946    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:05.757079 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.757051    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhsft"
Jul 09 18:05:05.780379 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.780190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jnp7h"
Jul 09 18:05:05.818845 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.818768    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b74d8bc8-dg69n"]
Jul 09 18:05:05.858609 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.857795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:05.905811 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.905427    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zrxtr"]
Jul 09 18:05:05.908058 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:05.908031    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40f86e5_325b_4c1f_8468_59b44d58b0c4.slice/crio-b1c25a4ca5d06335c0b51313146f785ec83ac4a813e3a061769c105130fc0e96 WatchSource:0}: Error finding container b1c25a4ca5d06335c0b51313146f785ec83ac4a813e3a061769c105130fc0e96: Status 404 returned error can't find the container with id b1c25a4ca5d06335c0b51313146f785ec83ac4a813e3a061769c105130fc0e96
Jul 09 18:05:05.942585 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.942542    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhsft"]
Jul 09 18:05:05.944649 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:05.944613    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f45cd8f_7df7_4798_a79e_b292782eef6f.slice/crio-8ca36a05e5e4c8c3bba38f47c622886c62ea81d1f5687f5e92e3e357d3b71c4f WatchSource:0}: Error finding container 8ca36a05e5e4c8c3bba38f47c622886c62ea81d1f5687f5e92e3e357d3b71c4f: Status 404 returned error can't find the container with id 8ca36a05e5e4c8c3bba38f47c622886c62ea81d1f5687f5e92e3e357d3b71c4f
Jul 09 18:05:05.965199 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:05.965172    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jnp7h"]
Jul 09 18:05:05.968598 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:05.968573    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e494343_ee4c_4a4f_8989_24a3e4bda412.slice/crio-c4891776761df36c05ec70737d6b81a591f9a6bbb93333e5be655c83314d1752 WatchSource:0}: Error finding container c4891776761df36c05ec70737d6b81a591f9a6bbb93333e5be655c83314d1752: Status 404 returned error can't find the container with id c4891776761df36c05ec70737d6b81a591f9a6bbb93333e5be655c83314d1752
Jul 09 18:05:06.286154 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.286124    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:06.368194 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.368159    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:06.383836 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.383801    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:05:06.383997 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.383967    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:05:06.389538 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.389472    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:06.389538 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.389517    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:06.389746 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.389493    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-skcpw\""
Jul 09 18:05:06.389832 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.389811    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:06.590605 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.590520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrxtr" event={"ID":"d40f86e5-325b-4c1f-8468-59b44d58b0c4","Type":"ContainerStarted","Data":"b1c25a4ca5d06335c0b51313146f785ec83ac4a813e3a061769c105130fc0e96"}
Jul 09 18:05:06.593070 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.593038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n" event={"ID":"1e0cc46f-0a9d-4139-adb0-969e5056fe79","Type":"ContainerStarted","Data":"a62eb98402e54c533d5b5eaa2bf74fe31b1124200377a6b8a8dd83e658307613"}
Jul 09 18:05:06.593212 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.593096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n" event={"ID":"1e0cc46f-0a9d-4139-adb0-969e5056fe79","Type":"ContainerStarted","Data":"96ee06d7d11fadd011c6e9c381ed3b0b12698c28772386e5b7c476877d3af3bb"}
Jul 09 18:05:06.593266 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.593223    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:06.594603 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.594577    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhsft" event={"ID":"0f45cd8f-7df7-4798-a79e-b292782eef6f","Type":"ContainerStarted","Data":"8ca36a05e5e4c8c3bba38f47c622886c62ea81d1f5687f5e92e3e357d3b71c4f"}
Jul 09 18:05:06.596085 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.596032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jnp7h" event={"ID":"4e494343-ee4c-4a4f-8989-24a3e4bda412","Type":"ContainerStarted","Data":"c4891776761df36c05ec70737d6b81a591f9a6bbb93333e5be655c83314d1752"}
Jul 09 18:05:06.632234 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:06.631218    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n" podStartSLOduration=2.631197388 podStartE2EDuration="2.631197388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:06.631112353 +0000 UTC m=+50.782749628" watchObservedRunningTime="2026-07-09 18:05:06.631197388 +0000 UTC m=+50.782834666"
Jul 09 18:05:09.607027 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:09.606986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrxtr" event={"ID":"d40f86e5-325b-4c1f-8468-59b44d58b0c4","Type":"ContainerStarted","Data":"aa4b2c3c4e5a393347502cbe991c833be4d6fac37bf0e212b29507cf9c7fa4d2"}
Jul 09 18:05:10.823463 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:10.823422    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:10.899295 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:10.899245    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:12.615212 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:12.615172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhsft" event={"ID":"0f45cd8f-7df7-4798-a79e-b292782eef6f","Type":"ContainerStarted","Data":"5b667deae69358cb209bc77b229644afe70774f4ba3961800f22a7527fbf3446"}
Jul 09 18:05:12.616949 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:12.616924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jnp7h" event={"ID":"4e494343-ee4c-4a4f-8989-24a3e4bda412","Type":"ContainerStarted","Data":"b3d43cc4c216bd37a5a24d3439d6ac1c5ef798a3eb282dfc2001d71028305849"}
Jul 09 18:05:12.617024 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:12.616982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jnp7h" event={"ID":"4e494343-ee4c-4a4f-8989-24a3e4bda412","Type":"ContainerStarted","Data":"e48f1466823045354697a5abc5d38f6418eb224b7abd34b5d691b810d7f7bcd0"}
Jul 09 18:05:12.618580 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:12.618559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrxtr" event={"ID":"d40f86e5-325b-4c1f-8468-59b44d58b0c4","Type":"ContainerStarted","Data":"83ffc486d3c58a3a195b8a8a246d5f4fb21fd4c748873f01c86f4ffe0c80616f"}
Jul 09 18:05:12.618693 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:12.618681    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:12.635669 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:12.635629    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mhsft" podStartSLOduration=1.79404081 podStartE2EDuration="7.635616946s" totalImagesPullingTime="5.841576136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:12.635347507 +0000 UTC m=+56.786984778" watchObservedRunningTime="2026-07-09 18:05:12.635616946 +0000 UTC m=+56.787254218"
Jul 09 18:05:12.663290 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:12.663236    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zrxtr" podStartSLOduration=4.619697638 podStartE2EDuration="7.663216957s" totalImagesPullingTime="3.043519319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:12.662973678 +0000 UTC m=+56.814610950" watchObservedRunningTime="2026-07-09 18:05:12.663216957 +0000 UTC m=+56.814854232"
Jul 09 18:05:12.898173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:12.898130    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:12.999764 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:12.999728    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:14.999308 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:14.999267    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.100493 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.100453    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.208421 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.208388    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.317254 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.317168    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.460223 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.460190    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.630701 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.630617    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jnp7h" event={"ID":"4e494343-ee4c-4a4f-8989-24a3e4bda412","Type":"ContainerStarted","Data":"e1926c433403aa1a84139d3ee98c37cc29f58ecf339051ad95821c41598a23f3"}
Jul 09 18:05:15.669061 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.669024    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.710256 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.710195    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jnp7h" podStartSLOduration=1.720063297 podStartE2EDuration="10.710178404s" totalImagesPullingTime="8.990115107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:15.699696033 +0000 UTC m=+59.851333305" watchObservedRunningTime="2026-07-09 18:05:15.710178404 +0000 UTC m=+59.861815675"
Jul 09 18:05:15.736125 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.736095    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp"]
Jul 09 18:05:15.738170 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.738154    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp"
Jul 09 18:05:15.749342 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.749321    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zm8xq\""
Jul 09 18:05:15.749997 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.749983    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 18:05:15.754078 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.754064    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 18:05:15.780750 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.780726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccv7\" (UniqueName: \"kubernetes.io/projected/14651ff7-a827-44f9-9e24-612d360786cf-kube-api-access-gccv7\") pod \"migrator-7f59b54f6c-tjczp\" (UID: \"14651ff7-a827-44f9-9e24-612d360786cf\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp"
Jul 09 18:05:15.803216 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.803157    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp"]
Jul 09 18:05:15.879466 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.879427    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.881407 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.881363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccv7\" (UniqueName: \"kubernetes.io/projected/14651ff7-a827-44f9-9e24-612d360786cf-kube-api-access-gccv7\") pod \"migrator-7f59b54f6c-tjczp\" (UID: \"14651ff7-a827-44f9-9e24-612d360786cf\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp"
Jul 09 18:05:15.897157 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.897132    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccv7\" (UniqueName: \"kubernetes.io/projected/14651ff7-a827-44f9-9e24-612d360786cf-kube-api-access-gccv7\") pod \"migrator-7f59b54f6c-tjczp\" (UID: \"14651ff7-a827-44f9-9e24-612d360786cf\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp"
Jul 09 18:05:15.989461 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:15.989427    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:16.046386 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:16.046351    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp"
Jul 09 18:05:16.064729 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:16.064695    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:16.170841 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:16.170758    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp"]
Jul 09 18:05:16.173282 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:16.173256    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14651ff7_a827_44f9_9e24_612d360786cf.slice/crio-b81a5dad5924eb3b7cce413030fba8121cc9bd2fcf0c31db0fd4d725416354ba WatchSource:0}: Error finding container b81a5dad5924eb3b7cce413030fba8121cc9bd2fcf0c31db0fd4d725416354ba: Status 404 returned error can't find the container with id b81a5dad5924eb3b7cce413030fba8121cc9bd2fcf0c31db0fd4d725416354ba
Jul 09 18:05:16.403168 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:16.403118    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:16.636323 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:16.636282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp" event={"ID":"14651ff7-a827-44f9-9e24-612d360786cf","Type":"ContainerStarted","Data":"b81a5dad5924eb3b7cce413030fba8121cc9bd2fcf0c31db0fd4d725416354ba"}
Jul 09 18:05:16.988084 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:16.987988    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.074063 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:17.074029    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.209817 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:17.209780    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.343872 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:17.343837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.542920 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:17.542897    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7npv"
Jul 09 18:05:17.640741 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:17.640709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp" event={"ID":"14651ff7-a827-44f9-9e24-612d360786cf","Type":"ContainerStarted","Data":"0742d19688c9a1525a8d2cd652b35295969d13830ead2b49077a6c14ec326b6c"}
Jul 09 18:05:17.793983 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:17.793947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.892330 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:17.892234    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.990921 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:17.990890    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:18.646165 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:18.646130    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp" event={"ID":"14651ff7-a827-44f9-9e24-612d360786cf","Type":"ContainerStarted","Data":"8ccb9b6bc4cceecbf40083f25f1a5830da5c589966b93ed22db745bc5aecc3d9"}
Jul 09 18:05:18.669657 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:18.669605    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tjczp" podStartSLOduration=2.375546643 podStartE2EDuration="3.669589653s" totalImagesPullingTime="1.29404301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:18.668472706 +0000 UTC m=+62.820109978" watchObservedRunningTime="2026-07-09 18:05:18.669589653 +0000 UTC m=+62.821226924"
Jul 09 18:05:18.770067 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:18.770023    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:18.871464 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:18.871427    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:18.967163 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:18.967083    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:19.624049 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:19.624017    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zrxtr"
Jul 09 18:05:20.799507 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:20.799462    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:20.884862 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:20.884831    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:21.199923 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.199841    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:21.485814 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.485742    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:21.665167 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.665137    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:21.785830 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.785805    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9f4bn"]
Jul 09 18:05:21.788175 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.788161    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.792058 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.792041    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:05:21.792786 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.792627    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:05:21.792786 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.792677    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:05:21.792786 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.792715    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:05:21.794008 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.793987    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:05:21.794424 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.794407    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:05:21.794798 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.794779    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mtfc8\""
Jul 09 18:05:21.797059 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.797027    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:21.825492 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.825457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.825768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.825510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-tls\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.825768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.825537    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d3414c2-9767-4569-bb59-23f3f52d13a9-root\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.825768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.825626    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-wtmp\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.825768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.825677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d3414c2-9767-4569-bb59-23f3f52d13a9-sys\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.825768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.825702    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-textfile\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.825768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.825732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.825768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.825761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d3414c2-9767-4569-bb59-23f3f52d13a9-metrics-client-ca\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.825987 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.825778    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-794mk\" (UniqueName: \"kubernetes.io/projected/1d3414c2-9767-4569-bb59-23f3f52d13a9-kube-api-access-794mk\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.915118 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.915090    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:21.927066 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-wtmp\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927188 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d3414c2-9767-4569-bb59-23f3f52d13a9-sys\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927188 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-textfile\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927188 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927188 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927144    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d3414c2-9767-4569-bb59-23f3f52d13a9-metrics-client-ca\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927188 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-794mk\" (UniqueName: \"kubernetes.io/projected/1d3414c2-9767-4569-bb59-23f3f52d13a9-kube-api-access-794mk\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927188 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d3414c2-9767-4569-bb59-23f3f52d13a9-sys\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927457 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927457 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927209    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-wtmp\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927457 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-tls\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927457 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d3414c2-9767-4569-bb59-23f3f52d13a9-root\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927457 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d3414c2-9767-4569-bb59-23f3f52d13a9-root\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927457 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:21.927407    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 18:05:21.927457 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927415    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-textfile\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927712 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:05:21.927512    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-tls podName:1d3414c2-9767-4569-bb59-23f3f52d13a9 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:22.427464345 +0000 UTC m=+66.579101600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-tls") pod "node-exporter-9f4bn" (UID: "1d3414c2-9767-4569-bb59-23f3f52d13a9") : secret "node-exporter-tls" not found
Jul 09 18:05:21.927782 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d3414c2-9767-4569-bb59-23f3f52d13a9-metrics-client-ca\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.927817 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.927772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.929359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.929342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:21.950706 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:21.950684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-794mk\" (UniqueName: \"kubernetes.io/projected/1d3414c2-9767-4569-bb59-23f3f52d13a9-kube-api-access-794mk\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:22.015505 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.015467    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:22.027595 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.027569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:05:22.030378 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.030364    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:22.040715 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.040664    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1803cc34-6cae-4580-a034-b67c1cab8b8e-metrics-certs\") pod \"network-metrics-daemon-zsbvk\" (UID: \"1803cc34-6cae-4580-a034-b67c1cab8b8e\") " pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:05:22.128778 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.128750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:05:22.131751 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.131737    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:22.141182 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.141158    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8-original-pull-secret\") pod \"global-pull-secret-syncer-5wc7j\" (UID: \"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8\") " pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:05:22.198512 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.198491    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mw97p\""
Jul 09 18:05:22.205323 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.205303    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsbvk"
Jul 09 18:05:22.230064 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.230037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:05:22.233072 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.233056    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:22.243351 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.243334    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:22.253349 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.253330    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xknv5\" (UniqueName: \"kubernetes.io/projected/882b7841-7d01-46cb-afca-f21a77389624-kube-api-access-xknv5\") pod \"network-check-target-gbg4t\" (UID: \"882b7841-7d01-46cb-afca-f21a77389624\") " pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:05:22.258449 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.258423    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:22.295901 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.295843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wc7j"
Jul 09 18:05:22.337739 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.337715    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-skcpw\""
Jul 09 18:05:22.344574 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.344550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:05:22.354582 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.354405    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zsbvk"]
Jul 09 18:05:22.358850 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:22.358816    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1803cc34_6cae_4580_a034_b67c1cab8b8e.slice/crio-193e3d44a733fe1db47682927ae7ecbe940ea6d2e62da3319af0f1c57b7dbd29 WatchSource:0}: Error finding container 193e3d44a733fe1db47682927ae7ecbe940ea6d2e62da3319af0f1c57b7dbd29: Status 404 returned error can't find the container with id 193e3d44a733fe1db47682927ae7ecbe940ea6d2e62da3319af0f1c57b7dbd29
Jul 09 18:05:22.404291 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.404222    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:22.431997 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.431973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-tls\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:22.434582 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.434557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d3414c2-9767-4569-bb59-23f3f52d13a9-node-exporter-tls\") pod \"node-exporter-9f4bn\" (UID: \"1d3414c2-9767-4569-bb59-23f3f52d13a9\") " pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:22.452756 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.452732    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5wc7j"]
Jul 09 18:05:22.455276 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:22.455248    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19aec6c_4cdd_41dc_b6b5_3b5adfe023a8.slice/crio-0617db8973554201d522fe240118e35439064e1c52d8087b2812b159d04ff488 WatchSource:0}: Error finding container 0617db8973554201d522fe240118e35439064e1c52d8087b2812b159d04ff488: Status 404 returned error can't find the container with id 0617db8973554201d522fe240118e35439064e1c52d8087b2812b159d04ff488
Jul 09 18:05:22.478944 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.478920    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbg4t"]
Jul 09 18:05:22.481283 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:22.481264    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882b7841_7d01_46cb_afca_f21a77389624.slice/crio-9b28a8fd015e23372875e87862a39d56e329a77fe51020c171cdaa9346e69e1d WatchSource:0}: Error finding container 9b28a8fd015e23372875e87862a39d56e329a77fe51020c171cdaa9346e69e1d: Status 404 returned error can't find the container with id 9b28a8fd015e23372875e87862a39d56e329a77fe51020c171cdaa9346e69e1d
Jul 09 18:05:22.533840 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.533809    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:22.659264 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.659206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zsbvk" event={"ID":"1803cc34-6cae-4580-a034-b67c1cab8b8e","Type":"ContainerStarted","Data":"193e3d44a733fe1db47682927ae7ecbe940ea6d2e62da3319af0f1c57b7dbd29"}
Jul 09 18:05:22.660452 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.660430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbg4t" event={"ID":"882b7841-7d01-46cb-afca-f21a77389624","Type":"ContainerStarted","Data":"9b28a8fd015e23372875e87862a39d56e329a77fe51020c171cdaa9346e69e1d"}
Jul 09 18:05:22.661444 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.661419    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5wc7j" event={"ID":"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8","Type":"ContainerStarted","Data":"0617db8973554201d522fe240118e35439064e1c52d8087b2812b159d04ff488"}
Jul 09 18:05:22.665781 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.665760    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:22.699732 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.699714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9f4bn"
Jul 09 18:05:22.710752 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:22.710733    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3414c2_9767_4569_bb59_23f3f52d13a9.slice/crio-4f808c9d115b8ebe36702e6ac55590c557b8c4b437e61ab25fcb745c74e34657 WatchSource:0}: Error finding container 4f808c9d115b8ebe36702e6ac55590c557b8c4b437e61ab25fcb745c74e34657: Status 404 returned error can't find the container with id 4f808c9d115b8ebe36702e6ac55590c557b8c4b437e61ab25fcb745c74e34657
Jul 09 18:05:22.777888 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.777866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:22.844475 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.844447    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:05:22.847153 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.847134    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.850273 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.850245    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 09 18:05:22.850410 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.850257    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 09 18:05:22.850573 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.850554    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 09 18:05:22.850689 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.850558    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 09 18:05:22.851581 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.851550    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 09 18:05:22.851783 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.851767    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 09 18:05:22.852053 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.852021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 09 18:05:22.852053 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.852042    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 09 18:05:22.856066 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.856047    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-w6gq4\""
Jul 09 18:05:22.859271 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.859253    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 09 18:05:22.871895 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.871814    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:05:22.883537 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.883496    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:22.935029 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.934974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935029 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935181 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935181 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-web-config\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935181 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2h7p\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-kube-api-access-n2h7p\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935181 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935164    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935396 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935396 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935237    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935396 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935396 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935314    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935396 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935396 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.935718 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.935396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-out\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.969792 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:22.969749    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:23.036658 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.036767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.036767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-web-config\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.036767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2h7p\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-kube-api-access-n2h7p\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.036767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.036767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.037030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.037030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.037030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.037030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.037030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.037030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.036949    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-out\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.037030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.037027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.037469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.037437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.038642 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.038617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.039823 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.039778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.040218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.040193    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.041071 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.041044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.041625 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.041602    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-web-config\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.041860 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.041668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.041860 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.041674    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.042140 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.042088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.042276 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.042256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.042475 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.042455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.043075 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.043020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-out\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.053641 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.053619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2h7p\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-kube-api-access-n2h7p\") pod \"alertmanager-main-0\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.158804 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.158765    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:23.508444 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.508414    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:05:23.513042 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:23.513011    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cec2df_4e4f_4fa1_925a_885af3c70de9.slice/crio-ee3e5a11101cd896236cb369f1a0ae471184e838d73a92c5e7509279040b701f WatchSource:0}: Error finding container ee3e5a11101cd896236cb369f1a0ae471184e838d73a92c5e7509279040b701f: Status 404 returned error can't find the container with id ee3e5a11101cd896236cb369f1a0ae471184e838d73a92c5e7509279040b701f
Jul 09 18:05:23.666889 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.666749    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerStarted","Data":"ee3e5a11101cd896236cb369f1a0ae471184e838d73a92c5e7509279040b701f"}
Jul 09 18:05:23.668606 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.668531    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9f4bn" event={"ID":"1d3414c2-9767-4569-bb59-23f3f52d13a9","Type":"ContainerStarted","Data":"4f808c9d115b8ebe36702e6ac55590c557b8c4b437e61ab25fcb745c74e34657"}
Jul 09 18:05:23.671079 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.671051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zsbvk" event={"ID":"1803cc34-6cae-4580-a034-b67c1cab8b8e","Type":"ContainerStarted","Data":"4b383665a00d5adcce0fc55637bb993c66c2d66f84b9867f3251f8ab28d8e411"}
Jul 09 18:05:23.671197 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.671099    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zsbvk" event={"ID":"1803cc34-6cae-4580-a034-b67c1cab8b8e","Type":"ContainerStarted","Data":"8e2310ebea306c0539390b5dd216c0d99af49d7a70623e7834495ce23e79630f"}
Jul 09 18:05:23.700014 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.699964    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zsbvk" podStartSLOduration=66.69472057 podStartE2EDuration="1m7.699944823s" totalImagesPullingTime="1.005224253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:23.697612471 +0000 UTC m=+67.849249743" watchObservedRunningTime="2026-07-09 18:05:23.699944823 +0000 UTC m=+67.851582098"
Jul 09 18:05:23.830744 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.830710    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:23.929349 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:23.929234    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:24.523473 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.523369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:24.628095 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.628058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:24.676826 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.676793    2560 generic.go:356] "Generic (PLEG): container finished" podID="1d3414c2-9767-4569-bb59-23f3f52d13a9" containerID="b0a77f6f3af4a90ca788632e293346639e885ca6525cc5941c36c535a8319a85" exitCode=0
Jul 09 18:05:24.676976 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.676870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9f4bn" event={"ID":"1d3414c2-9767-4569-bb59-23f3f52d13a9","Type":"ContainerDied","Data":"b0a77f6f3af4a90ca788632e293346639e885ca6525cc5941c36c535a8319a85"}
Jul 09 18:05:24.721265 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.721237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:24.804989 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.804963    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f87cdc4c-998tg"]
Jul 09 18:05:24.828164 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.828069    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f87cdc4c-998tg"]
Jul 09 18:05:24.828783 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.828117    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:24.832190 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.832170    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j6lb2\""
Jul 09 18:05:24.832283 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.832262    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 09 18:05:24.832611 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.832593    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 09 18:05:24.832731 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.832625    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1hmjeo31l61ja\""
Jul 09 18:05:24.835415 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.834088    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 09 18:05:24.835415 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.834340    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 09 18:05:24.835415 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.835258    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 09 18:05:24.953549 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.953513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-grpc-tls\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:24.953934 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.953576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:24.953934 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.953614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:24.953934 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.953655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:24.953934 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.953680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttr5\" (UniqueName: \"kubernetes.io/projected/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-kube-api-access-tttr5\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:24.953934 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.953715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-tls\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:24.953934 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.953749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-metrics-client-ca\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:24.953934 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.953791    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:24.976701 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:24.976667    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:25.054862 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.054783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.054862 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.054838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-grpc-tls\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.055081 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.054894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.055081 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.054931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.055184 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.055100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.055184 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.055140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttr5\" (UniqueName: \"kubernetes.io/projected/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-kube-api-access-tttr5\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.055292 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.055181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-tls\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.055292 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.055223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-metrics-client-ca\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.056001 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.055948    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-metrics-client-ca\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.058206 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.058179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-tls\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.058880 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.058838    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.058964 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.058921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.058964 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.058940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.059911 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.059888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-grpc-tls\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.060364 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.060327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.066819 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.066799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttr5\" (UniqueName: \"kubernetes.io/projected/9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5-kube-api-access-tttr5\") pod \"thanos-querier-6f87cdc4c-998tg\" (UID: \"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5\") " pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.148350 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.148303    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:25.255018 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.254981    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:25.364321 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.364215    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:25.687326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.687236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9f4bn" event={"ID":"1d3414c2-9767-4569-bb59-23f3f52d13a9","Type":"ContainerStarted","Data":"dd3cc24429cec2d77df57cbfd4a4d0e9bd2edff8ffa82b869e28180f84adc990"}
Jul 09 18:05:25.687511 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.687333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9f4bn" event={"ID":"1d3414c2-9767-4569-bb59-23f3f52d13a9","Type":"ContainerStarted","Data":"073261083c559013e8a9fdb1775741cadb85fab2188f049bc0cbe919097f4ac8"}
Jul 09 18:05:25.712930 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:25.712877    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9f4bn" podStartSLOduration=3.236130195 podStartE2EDuration="4.712860118s" totalImagesPullingTime="1.398244545s" totalInitContainerRuntime="78.485378ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:25.712115884 +0000 UTC m=+69.863753157" watchObservedRunningTime="2026-07-09 18:05:25.712860118 +0000 UTC m=+69.864497369"
Jul 09 18:05:26.196985 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.196956    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:26.272275 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.272241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:26.339975 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.339933    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:26.398910 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.398881    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d5c4444c8-4qwcq"]
Jul 09 18:05:26.401088 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.401068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.407196 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.407159    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 18:05:26.408589 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.408561    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-sqjnt\""
Jul 09 18:05:26.408672 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.408583    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 09 18:05:26.408972 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.408821    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 09 18:05:26.408972 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.408879    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 09 18:05:26.408972 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.408879    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3r3h8s3thidfv\""
Jul 09 18:05:26.418646 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.418623    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d5c4444c8-4qwcq"]
Jul 09 18:05:26.466766 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.466703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4ec1d066-e324-4326-bf9b-d002fcdb3f80-metrics-server-audit-profiles\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.466766 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.466754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4ec1d066-e324-4326-bf9b-d002fcdb3f80-secret-metrics-server-tls\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.466961 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.466783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec1d066-e324-4326-bf9b-d002fcdb3f80-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.466961 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.466809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4ec1d066-e324-4326-bf9b-d002fcdb3f80-audit-log\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.466961 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.466830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4ec1d066-e324-4326-bf9b-d002fcdb3f80-client-ca-bundle\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.466961 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.466873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4ec1d066-e324-4326-bf9b-d002fcdb3f80-secret-metrics-server-client-certs\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.466961 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.466904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6kz\" (UniqueName: \"kubernetes.io/projected/4ec1d066-e324-4326-bf9b-d002fcdb3f80-kube-api-access-sn6kz\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.568044 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.568009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4ec1d066-e324-4326-bf9b-d002fcdb3f80-metrics-server-audit-profiles\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.568158 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.568078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4ec1d066-e324-4326-bf9b-d002fcdb3f80-secret-metrics-server-tls\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.568158 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.568107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec1d066-e324-4326-bf9b-d002fcdb3f80-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.568158 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.568140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4ec1d066-e324-4326-bf9b-d002fcdb3f80-audit-log\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.568311 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.568165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4ec1d066-e324-4326-bf9b-d002fcdb3f80-client-ca-bundle\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.568311 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.568207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4ec1d066-e324-4326-bf9b-d002fcdb3f80-secret-metrics-server-client-certs\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.568311 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.568236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6kz\" (UniqueName: \"kubernetes.io/projected/4ec1d066-e324-4326-bf9b-d002fcdb3f80-kube-api-access-sn6kz\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.568748 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.568714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4ec1d066-e324-4326-bf9b-d002fcdb3f80-audit-log\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.568883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.568859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec1d066-e324-4326-bf9b-d002fcdb3f80-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.569136 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.569113    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4ec1d066-e324-4326-bf9b-d002fcdb3f80-metrics-server-audit-profiles\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.571281 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.571258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4ec1d066-e324-4326-bf9b-d002fcdb3f80-secret-metrics-server-tls\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.571357 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.571325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4ec1d066-e324-4326-bf9b-d002fcdb3f80-secret-metrics-server-client-certs\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.571357 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.571354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4ec1d066-e324-4326-bf9b-d002fcdb3f80-client-ca-bundle\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.577497 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.577452    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6kz\" (UniqueName: \"kubernetes.io/projected/4ec1d066-e324-4326-bf9b-d002fcdb3f80-kube-api-access-sn6kz\") pod \"metrics-server-d5c4444c8-4qwcq\" (UID: \"4ec1d066-e324-4326-bf9b-d002fcdb3f80\") " pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:26.711394 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:26.711368    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:27.088514 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:27.088466    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:27.215640 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:27.215609    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:27.605001 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:27.604979    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b74d8bc8-dg69n"
Jul 09 18:05:27.784687 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:27.784661    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d5c4444c8-4qwcq"]
Jul 09 18:05:27.785229 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:27.785202    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec1d066_e324_4326_bf9b_d002fcdb3f80.slice/crio-922932d84f858ebb404da84c0bdb7b099b1e439563bef320a0dd1c1ef173ca4c WatchSource:0}: Error finding container 922932d84f858ebb404da84c0bdb7b099b1e439563bef320a0dd1c1ef173ca4c: Status 404 returned error can't find the container with id 922932d84f858ebb404da84c0bdb7b099b1e439563bef320a0dd1c1ef173ca4c
Jul 09 18:05:27.806604 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:27.806566    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6b00d6_56a4_4e47_bb07_8fae0fafa5d5.slice/crio-c788ea0aad76bfd84af7bb71f794b6c005f849b96f8c49803edee40f8fa1391c WatchSource:0}: Error finding container c788ea0aad76bfd84af7bb71f794b6c005f849b96f8c49803edee40f8fa1391c: Status 404 returned error can't find the container with id c788ea0aad76bfd84af7bb71f794b6c005f849b96f8c49803edee40f8fa1391c
Jul 09 18:05:27.806945 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:27.806905    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f87cdc4c-998tg"]
Jul 09 18:05:27.890820 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:27.890741    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.131502 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.131455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.248927 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.248895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.431421 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.431371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.539303 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.539225    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.705109 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.705075    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq" event={"ID":"4ec1d066-e324-4326-bf9b-d002fcdb3f80","Type":"ContainerStarted","Data":"922932d84f858ebb404da84c0bdb7b099b1e439563bef320a0dd1c1ef173ca4c"}
Jul 09 18:05:28.707112 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.707084    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerID="9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed" exitCode=0
Jul 09 18:05:28.707259 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.707180    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerDied","Data":"9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed"}
Jul 09 18:05:28.709231 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.709209    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbg4t" event={"ID":"882b7841-7d01-46cb-afca-f21a77389624","Type":"ContainerStarted","Data":"8500d4737432abd39f3849093887f87dcd8902e73a528ee2f412c241a54de324"}
Jul 09 18:05:28.709364 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.709348    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:05:28.711839 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.711764    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg" event={"ID":"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5","Type":"ContainerStarted","Data":"c788ea0aad76bfd84af7bb71f794b6c005f849b96f8c49803edee40f8fa1391c"}
Jul 09 18:05:28.713388 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.713366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5wc7j" event={"ID":"c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8","Type":"ContainerStarted","Data":"160bbf429dce96b6c7e308986173abc15e60edf6a7d16e47b93f1e8f27e949e2"}
Jul 09 18:05:28.812430 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.812344    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gbg4t" podStartSLOduration=66.700039637 podStartE2EDuration="1m11.812330043s" totalImagesPullingTime="5.112290406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:28.791086941 +0000 UTC m=+72.942724214" watchObservedRunningTime="2026-07-09 18:05:28.812330043 +0000 UTC m=+72.963967317"
Jul 09 18:05:28.813624 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.813508    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5wc7j" podStartSLOduration=66.670017516 podStartE2EDuration="1m11.813470329s" totalImagesPullingTime="5.143452813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:28.812201383 +0000 UTC m=+72.963838658" watchObservedRunningTime="2026-07-09 18:05:28.813470329 +0000 UTC m=+72.965107601"
Jul 09 18:05:28.876991 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:28.876944    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.719760 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:29.719690    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq" event={"ID":"4ec1d066-e324-4326-bf9b-d002fcdb3f80","Type":"ContainerStarted","Data":"d30d646d55fadcaac466e378f3eaf25b0f8502b37c3f35f81d229528326efdb6"}
Jul 09 18:05:29.723280 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:29.722869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg" event={"ID":"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5","Type":"ContainerStarted","Data":"5862ba0ea4651c2760fca6e243980092697a01b32a9db37f6c3de8e456aa82a4"}
Jul 09 18:05:29.748589 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:29.748547    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq" podStartSLOduration=1.964300535 podStartE2EDuration="3.748531861s" totalImagesPullingTime="1.784231326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:29.746982608 +0000 UTC m=+73.898619881" watchObservedRunningTime="2026-07-09 18:05:29.748531861 +0000 UTC m=+73.900169111"
Jul 09 18:05:30.729329 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:30.729300    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerStarted","Data":"72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d"}
Jul 09 18:05:30.729634 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:30.729370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerStarted","Data":"0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702"}
Jul 09 18:05:30.729634 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:30.729395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerStarted","Data":"8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0"}
Jul 09 18:05:30.729634 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:30.729418    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerStarted","Data":"cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d"}
Jul 09 18:05:30.731977 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:30.731944    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg" event={"ID":"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5","Type":"ContainerStarted","Data":"98a3c44eb8388407327f7d1acc136b32a874718271426e92680e0950da39bb93"}
Jul 09 18:05:30.732079 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:30.732024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg" event={"ID":"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5","Type":"ContainerStarted","Data":"524d79a27eafb4e22db10b3a4be9272f9dc708ba25382938d89f9158dd66d8b2"}
Jul 09 18:05:31.740080 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.740044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerStarted","Data":"c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d"}
Jul 09 18:05:31.740496 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.740133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerStarted","Data":"7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d"}
Jul 09 18:05:31.743358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.743332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg" event={"ID":"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5","Type":"ContainerStarted","Data":"62c9acfbc9f120b322a210730cd092e0443290e2aa0397e17fc18824b8e52305"}
Jul 09 18:05:31.743451 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.743394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg" event={"ID":"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5","Type":"ContainerStarted","Data":"0f8756d1efb7424a7e8b42f6ebfaa68a1ce9bce128bbff05d2e76cbbce480608"}
Jul 09 18:05:31.743451 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.743423    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg" event={"ID":"9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5","Type":"ContainerStarted","Data":"061e89809fb189fae42ff9211b4594d1cff3792db4271dd0af34d5292ee3cb50"}
Jul 09 18:05:31.743558 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.743473    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:31.772093 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.772047    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.9418332449999998 podStartE2EDuration="9.772031753s" totalImagesPullingTime="5.807135111s" totalInitContainerRuntime="23.063397ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:31.770075356 +0000 UTC m=+75.921712647" watchObservedRunningTime="2026-07-09 18:05:31.772031753 +0000 UTC m=+75.923669025"
Jul 09 18:05:31.813023 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.812964    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85bdc5b679-64nqh"]
Jul 09 18:05:31.816412 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.816371    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg" podStartSLOduration=4.982256358 podStartE2EDuration="7.816358343s" totalImagesPullingTime="2.834101985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:31.815137825 +0000 UTC m=+75.966775107" watchObservedRunningTime="2026-07-09 18:05:31.816358343 +0000 UTC m=+75.967995614"
Jul 09 18:05:31.816706 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.816688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:31.820716 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.820694    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 18:05:31.820803 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.820747    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 18:05:31.820861 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.820806    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 18:05:31.820861 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.820752    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6txwz\""
Jul 09 18:05:31.821195 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.821177    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 18:05:31.821286 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.821263    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 18:05:31.821558 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.821543    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 18:05:31.824099 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.824080    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 18:05:31.835425 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.835405    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85bdc5b679-64nqh"]
Jul 09 18:05:31.843466 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.843295    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 18:05:31.913427 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.913408    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-serving-cert\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:31.913524 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.913439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-oauth-config\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:31.913524 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.913463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-console-config\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:31.913524 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.913502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-service-ca\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:31.913660 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.913555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-trusted-ca-bundle\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:31.913660 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.913578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mc9d\" (UniqueName: \"kubernetes.io/projected/d26d248e-d37f-4d56-8606-4807f9068fb8-kube-api-access-5mc9d\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:31.913754 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:31.913667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-oauth-serving-cert\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.014684 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.014648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-serving-cert\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.014787 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.014698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-oauth-config\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.014787 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.014732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-console-config\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.014787 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.014769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-service-ca\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.014951 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.014808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-trusted-ca-bundle\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.014951 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.014832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mc9d\" (UniqueName: \"kubernetes.io/projected/d26d248e-d37f-4d56-8606-4807f9068fb8-kube-api-access-5mc9d\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.015041 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.014945    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-oauth-serving-cert\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.015616 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.015588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-service-ca\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.015737 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.015700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-console-config\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.015737 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.015706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-oauth-serving-cert\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.015737 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.015728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-trusted-ca-bundle\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.017749 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.017731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-oauth-config\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.017853 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.017833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-serving-cert\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.023514 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.023470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mc9d\" (UniqueName: \"kubernetes.io/projected/d26d248e-d37f-4d56-8606-4807f9068fb8-kube-api-access-5mc9d\") pod \"console-85bdc5b679-64nqh\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") " pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.127398 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.127365    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:32.193610 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.193583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:32.256347 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.256323    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85bdc5b679-64nqh"]
Jul 09 18:05:32.258210 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:32.258186    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26d248e_d37f_4d56_8606_4807f9068fb8.slice/crio-6d8b434c8d8ada4f55a62f4bb60c3a0e16c645b74594ba04a66bdc0831a3e637 WatchSource:0}: Error finding container 6d8b434c8d8ada4f55a62f4bb60c3a0e16c645b74594ba04a66bdc0831a3e637: Status 404 returned error can't find the container with id 6d8b434c8d8ada4f55a62f4bb60c3a0e16c645b74594ba04a66bdc0831a3e637
Jul 09 18:05:32.263702 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.263679    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:32.750537 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:32.750464    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bdc5b679-64nqh" event={"ID":"d26d248e-d37f-4d56-8606-4807f9068fb8","Type":"ContainerStarted","Data":"6d8b434c8d8ada4f55a62f4bb60c3a0e16c645b74594ba04a66bdc0831a3e637"}
Jul 09 18:05:33.612358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:33.612311    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:33.680741 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:33.680713    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:35.600151 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:35.600068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:35.663803 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:35.663777    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:35.764063 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:35.764038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bdc5b679-64nqh" event={"ID":"d26d248e-d37f-4d56-8606-4807f9068fb8","Type":"ContainerStarted","Data":"073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301"}
Jul 09 18:05:35.783453 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:35.783407    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85bdc5b679-64nqh" podStartSLOduration=1.794700578 podStartE2EDuration="4.783393065s" totalImagesPullingTime="2.988692487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:35.782770829 +0000 UTC m=+79.934408122" watchObservedRunningTime="2026-07-09 18:05:35.783393065 +0000 UTC m=+79.935030337"
Jul 09 18:05:37.757006 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:37.756974    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f87cdc4c-998tg"
Jul 09 18:05:38.326837 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:38.326797    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.391771 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:38.391744    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:39.298577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:39.298537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:39.398753 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:39.398704    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:39.467794 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:39.467762    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:42.127528 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.127495    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:42.127924 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.127550    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:42.132401 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.132376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:42.280606 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.280579    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85bdc5b679-64nqh"]
Jul 09 18:05:42.323545 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.323517    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-845b6dbd9d-c57dg"]
Jul 09 18:05:42.406933 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.406872    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845b6dbd9d-c57dg"]
Jul 09 18:05:42.407035 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.406979    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.492307 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.492281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-serving-cert\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.492428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.492311    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-service-ca\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.492428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.492332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-oauth-serving-cert\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.492428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.492352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwnx5\" (UniqueName: \"kubernetes.io/projected/2e9383d9-fd56-4987-833f-96f46c348eaa-kube-api-access-hwnx5\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.492625 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.492453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-console-config\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.492625 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.492535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-oauth-config\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.492625 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.492573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-trusted-ca-bundle\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.593612 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.593587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-oauth-serving-cert\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.593709 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.593619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwnx5\" (UniqueName: \"kubernetes.io/projected/2e9383d9-fd56-4987-833f-96f46c348eaa-kube-api-access-hwnx5\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.593709 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.593649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-console-config\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.593825 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.593765    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-oauth-config\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.593825 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.593798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-trusted-ca-bundle\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.593825 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.593822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-serving-cert\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.593964 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.593852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-service-ca\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.594314 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.594286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-oauth-serving-cert\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.594600 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.594575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-service-ca\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.594680 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.594596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-console-config\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.594680 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.594657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-trusted-ca-bundle\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.596467 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.596447    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-serving-cert\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.596534 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.596456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-oauth-config\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.607715 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.607696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwnx5\" (UniqueName: \"kubernetes.io/projected/2e9383d9-fd56-4987-833f-96f46c348eaa-kube-api-access-hwnx5\") pod \"console-845b6dbd9d-c57dg\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") " pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.715521 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.715461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:42.794277 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.794254    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:05:42.875580 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:42.875556    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845b6dbd9d-c57dg"]
Jul 09 18:05:42.878246 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:05:42.878214    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9383d9_fd56_4987_833f_96f46c348eaa.slice/crio-6cdded58b7e4fd02f7c7f7ae6ae1e67e4ec7cb41dac778bcf373468d72ed9ba2 WatchSource:0}: Error finding container 6cdded58b7e4fd02f7c7f7ae6ae1e67e4ec7cb41dac778bcf373468d72ed9ba2: Status 404 returned error can't find the container with id 6cdded58b7e4fd02f7c7f7ae6ae1e67e4ec7cb41dac778bcf373468d72ed9ba2
Jul 09 18:05:43.620963 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:43.620920    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.751831 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:43.751801    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.799191 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:43.799163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845b6dbd9d-c57dg" event={"ID":"2e9383d9-fd56-4987-833f-96f46c348eaa","Type":"ContainerStarted","Data":"7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb"}
Jul 09 18:05:43.799320 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:43.799237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845b6dbd9d-c57dg" event={"ID":"2e9383d9-fd56-4987-833f-96f46c348eaa","Type":"ContainerStarted","Data":"6cdded58b7e4fd02f7c7f7ae6ae1e67e4ec7cb41dac778bcf373468d72ed9ba2"}
Jul 09 18:05:43.843203 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:43.843161    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-845b6dbd9d-c57dg" podStartSLOduration=1.8431475389999998 podStartE2EDuration="1.843147539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:43.842016199 +0000 UTC m=+87.993653473" watchObservedRunningTime="2026-07-09 18:05:43.843147539 +0000 UTC m=+87.994784810"
Jul 09 18:05:45.546117 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:45.546084    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:45.611042 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:45.611010    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.712397 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:46.712365    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:46.712799 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:46.712411    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:05:47.185807 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:47.185768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.286401 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:47.286374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:52.716153 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:52.716108    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:52.716663 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:52.716210    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:52.721213 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:52.721176    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:52.834595 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:52.834572    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:05:53.037307 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:53.037277    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:53.141157 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:53.141126    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:53.238496 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:53.238445    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:55.119521 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:55.119469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:55.183317 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:55.183284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:59.725891 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:05:59.725857    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gbg4t"
Jul 09 18:06:06.716885 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:06.716850    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:06:06.720874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:06.720849    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d5c4444c8-4qwcq"
Jul 09 18:06:08.818519 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:08.818441    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85bdc5b679-64nqh" podUID="d26d248e-d37f-4d56-8606-4807f9068fb8" containerName="console" containerID="cri-o://073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301" gracePeriod=15
Jul 09 18:06:09.060410 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.060388    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:06:09.179461 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.179397    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-console-config\") pod \"d26d248e-d37f-4d56-8606-4807f9068fb8\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") "
Jul 09 18:06:09.179600 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.179471    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-oauth-serving-cert\") pod \"d26d248e-d37f-4d56-8606-4807f9068fb8\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") "
Jul 09 18:06:09.179600 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.179550    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-serving-cert\") pod \"d26d248e-d37f-4d56-8606-4807f9068fb8\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") "
Jul 09 18:06:09.179600 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.179577    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-service-ca\") pod \"d26d248e-d37f-4d56-8606-4807f9068fb8\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") "
Jul 09 18:06:09.179757 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.179602    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-oauth-config\") pod \"d26d248e-d37f-4d56-8606-4807f9068fb8\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") "
Jul 09 18:06:09.179757 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.179652    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mc9d\" (UniqueName: \"kubernetes.io/projected/d26d248e-d37f-4d56-8606-4807f9068fb8-kube-api-access-5mc9d\") pod \"d26d248e-d37f-4d56-8606-4807f9068fb8\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") "
Jul 09 18:06:09.179757 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.179680    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-trusted-ca-bundle\") pod \"d26d248e-d37f-4d56-8606-4807f9068fb8\" (UID: \"d26d248e-d37f-4d56-8606-4807f9068fb8\") "
Jul 09 18:06:09.179907 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.179878    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-console-config" (OuterVolumeSpecName: "console-config") pod "d26d248e-d37f-4d56-8606-4807f9068fb8" (UID: "d26d248e-d37f-4d56-8606-4807f9068fb8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:09.179960 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.179907    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d26d248e-d37f-4d56-8606-4807f9068fb8" (UID: "d26d248e-d37f-4d56-8606-4807f9068fb8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:09.180116 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.180080    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-service-ca" (OuterVolumeSpecName: "service-ca") pod "d26d248e-d37f-4d56-8606-4807f9068fb8" (UID: "d26d248e-d37f-4d56-8606-4807f9068fb8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:09.180206 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.180182    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d26d248e-d37f-4d56-8606-4807f9068fb8" (UID: "d26d248e-d37f-4d56-8606-4807f9068fb8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:09.181873 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.181844    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d26d248e-d37f-4d56-8606-4807f9068fb8" (UID: "d26d248e-d37f-4d56-8606-4807f9068fb8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:09.181958 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.181895    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d26d248e-d37f-4d56-8606-4807f9068fb8" (UID: "d26d248e-d37f-4d56-8606-4807f9068fb8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:09.181958 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.181912    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26d248e-d37f-4d56-8606-4807f9068fb8-kube-api-access-5mc9d" (OuterVolumeSpecName: "kube-api-access-5mc9d") pod "d26d248e-d37f-4d56-8606-4807f9068fb8" (UID: "d26d248e-d37f-4d56-8606-4807f9068fb8"). InnerVolumeSpecName "kube-api-access-5mc9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:09.281030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.281009    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mc9d\" (UniqueName: \"kubernetes.io/projected/d26d248e-d37f-4d56-8606-4807f9068fb8-kube-api-access-5mc9d\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:09.281030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.281029    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-trusted-ca-bundle\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:09.281154 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.281040    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-console-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:09.281154 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.281049    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-oauth-serving-cert\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:09.281154 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.281058    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-serving-cert\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:09.281154 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.281066    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d248e-d37f-4d56-8606-4807f9068fb8-service-ca\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:09.281154 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.281074    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d26d248e-d37f-4d56-8606-4807f9068fb8-console-oauth-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:09.895641 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.895604    2560 generic.go:356] "Generic (PLEG): container finished" podID="d26d248e-d37f-4d56-8606-4807f9068fb8" containerID="073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301" exitCode=0
Jul 09 18:06:09.895985 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.895668    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bdc5b679-64nqh"
Jul 09 18:06:09.895985 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.895668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bdc5b679-64nqh" event={"ID":"d26d248e-d37f-4d56-8606-4807f9068fb8","Type":"ContainerDied","Data":"073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301"}
Jul 09 18:06:09.895985 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.895721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bdc5b679-64nqh" event={"ID":"d26d248e-d37f-4d56-8606-4807f9068fb8","Type":"ContainerDied","Data":"6d8b434c8d8ada4f55a62f4bb60c3a0e16c645b74594ba04a66bdc0831a3e637"}
Jul 09 18:06:09.895985 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.895749    2560 scope.go:117] "RemoveContainer" containerID="073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301"
Jul 09 18:06:09.905659 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.905639    2560 scope.go:117] "RemoveContainer" containerID="073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301"
Jul 09 18:06:09.906019 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:06:09.905996    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301\": container with ID starting with 073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301 not found: ID does not exist" containerID="073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301"
Jul 09 18:06:09.906066 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.906027    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301"} err="failed to get container status \"073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301\": rpc error: code = NotFound desc = could not find container \"073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301\": container with ID starting with 073a37d370920ce382d0e473ff77a99d597ab2bf1aa9b5dc5f2fa1898b5ec301 not found: ID does not exist"
Jul 09 18:06:09.916277 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.916245    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85bdc5b679-64nqh"]
Jul 09 18:06:09.918681 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:09.918657    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85bdc5b679-64nqh"]
Jul 09 18:06:10.387924 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:10.387888    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d26d248e-d37f-4d56-8606-4807f9068fb8" path="/var/lib/kubelet/pods/d26d248e-d37f-4d56-8606-4807f9068fb8/volumes"
Jul 09 18:06:23.175459 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:23.175417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:25.790231 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:25.790178    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:25.855349 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:25.855311    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:37.767600 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:37.767548    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:37.834851 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:37.834807    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:39.227736 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.227703    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-845789c8d5-kxvv6"]
Jul 09 18:06:39.228095 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.227985    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26d248e-d37f-4d56-8606-4807f9068fb8" containerName="console"
Jul 09 18:06:39.228095 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.227998    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26d248e-d37f-4d56-8606-4807f9068fb8" containerName="console"
Jul 09 18:06:39.228095 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.228059    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26d248e-d37f-4d56-8606-4807f9068fb8" containerName="console"
Jul 09 18:06:39.229827 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.229806    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.241223 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.241197    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845789c8d5-kxvv6"]
Jul 09 18:06:39.285354 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.285324    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-service-ca\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.285454 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.285366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-oauth-config\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.285454 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.285386    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-oauth-serving-cert\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.285454 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.285415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76j48\" (UniqueName: \"kubernetes.io/projected/c4490d50-1057-4942-9ee2-847b334c3241-kube-api-access-76j48\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.285454 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.285445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-trusted-ca-bundle\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.285628 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.285463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-serving-cert\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.285628 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.285504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-console-config\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.298014 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.297989    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:39.386002 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.385973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-oauth-serving-cert\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.386112 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.386010    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76j48\" (UniqueName: \"kubernetes.io/projected/c4490d50-1057-4942-9ee2-847b334c3241-kube-api-access-76j48\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.386112 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.386044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-trusted-ca-bundle\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.386112 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.386071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-serving-cert\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.386112 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.386096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-console-config\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.386286 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.386158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-service-ca\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.386286 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.386196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-oauth-config\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.386923 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.386896    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-oauth-serving-cert\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.387077 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.387055    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-console-config\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.387152 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.387125    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-service-ca\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.387238 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.387221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-trusted-ca-bundle\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.388828 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.388797    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-serving-cert\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.388977 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.388952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-oauth-config\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.394617 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.394592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76j48\" (UniqueName: \"kubernetes.io/projected/c4490d50-1057-4942-9ee2-847b334c3241-kube-api-access-76j48\") pod \"console-845789c8d5-kxvv6\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") " pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.398422 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.398394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:39.455122 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.455091    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:39.539255 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.539227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:39.671567 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:39.671368    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845789c8d5-kxvv6"]
Jul 09 18:06:40.017258 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:40.017230    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:40.018425 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:40.018399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845789c8d5-kxvv6" event={"ID":"c4490d50-1057-4942-9ee2-847b334c3241","Type":"ContainerStarted","Data":"d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b"}
Jul 09 18:06:40.018551 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:40.018445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845789c8d5-kxvv6" event={"ID":"c4490d50-1057-4942-9ee2-847b334c3241","Type":"ContainerStarted","Data":"da6c3a4d45fafa18896d90e90ac183a8a98ae0bc2375d170d78ce0dcb2d825f4"}
Jul 09 18:06:40.039587 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:40.039539    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-845789c8d5-kxvv6" podStartSLOduration=1.039525531 podStartE2EDuration="1.039525531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:40.038563666 +0000 UTC m=+144.190200939" watchObservedRunningTime="2026-07-09 18:06:40.039525531 +0000 UTC m=+144.191162802"
Jul 09 18:06:40.117883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:40.117855    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:42.022721 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:42.022686    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:42.023160 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:42.023094    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="alertmanager" containerID="cri-o://cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d" gracePeriod=120
Jul 09 18:06:42.023260 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:42.023144    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy-metric" containerID="cri-o://7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d" gracePeriod=120
Jul 09 18:06:42.023260 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:42.023176    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy" containerID="cri-o://72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d" gracePeriod=120
Jul 09 18:06:42.023260 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:42.023202    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="config-reloader" containerID="cri-o://8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0" gracePeriod=120
Jul 09 18:06:42.023404 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:42.023180    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="prom-label-proxy" containerID="cri-o://c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d" gracePeriod=120
Jul 09 18:06:42.023404 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:42.023151    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy-web" containerID="cri-o://0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702" gracePeriod=120
Jul 09 18:06:43.036577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.036545    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerID="c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d" exitCode=0
Jul 09 18:06:43.036577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.036572    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerID="72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d" exitCode=0
Jul 09 18:06:43.036577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.036581    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerID="8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0" exitCode=0
Jul 09 18:06:43.037000 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.036592    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerID="cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d" exitCode=0
Jul 09 18:06:43.037000 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.036615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerDied","Data":"c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d"}
Jul 09 18:06:43.037000 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.036685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerDied","Data":"72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d"}
Jul 09 18:06:43.037000 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.036712    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerDied","Data":"8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0"}
Jul 09 18:06:43.037000 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.036737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerDied","Data":"cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d"}
Jul 09 18:06:43.255973 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.255953    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.313990 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.313927    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-main-tls\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.313990 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.313955    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-web\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.313990 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.313973    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-cluster-tls-config\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.313995    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314015    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-metrics-client-ca\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314035    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-tls-assets\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314066    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2h7p\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-kube-api-access-n2h7p\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314101    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-volume\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314124    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-out\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314150    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-web-config\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314171    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-trusted-ca-bundle\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314220    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314707 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314248    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-main-db\") pod \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\" (UID: \"f7cec2df-4e4f-4fa1-925a-885af3c70de9\") "
Jul 09 18:06:43.314707 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314389    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:43.314707 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.314553    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-metrics-client-ca\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.315031 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.315000    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:43.315746 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.315719    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:43.318449 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.318408    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.318557 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.318521    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-volume" (OuterVolumeSpecName: "config-volume") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.318647 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.318557    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-out" (OuterVolumeSpecName: "config-out") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:43.318755 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.318653    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-kube-api-access-n2h7p" (OuterVolumeSpecName: "kube-api-access-n2h7p") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "kube-api-access-n2h7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:43.318755 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.318718    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:43.318887 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.318782    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.318887 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.318870    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.319058 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.319041    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.323387 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.323359    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.328082 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.328061    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-web-config" (OuterVolumeSpecName: "web-config") pod "f7cec2df-4e4f-4fa1-925a-885af3c70de9" (UID: "f7cec2df-4e4f-4fa1-925a-885af3c70de9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.415317 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415293    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-main-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415317 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415315    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415325    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-cluster-tls-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415334    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415342    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-tls-assets\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415350    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2h7p\" (UniqueName: \"kubernetes.io/projected/f7cec2df-4e4f-4fa1-925a-885af3c70de9-kube-api-access-n2h7p\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415357    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-volume\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415365    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-config-out\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415372    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-web-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415379    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415387    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7cec2df-4e4f-4fa1-925a-885af3c70de9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.415428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:43.415395    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7cec2df-4e4f-4fa1-925a-885af3c70de9-alertmanager-main-db\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:06:44.044364 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.044325    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerID="7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d" exitCode=0
Jul 09 18:06:44.044364 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.044362    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerID="0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702" exitCode=0
Jul 09 18:06:44.044749 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.044417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerDied","Data":"7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d"}
Jul 09 18:06:44.044749 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.044449    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.044749 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.044509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerDied","Data":"0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702"}
Jul 09 18:06:44.044749 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.044542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7cec2df-4e4f-4fa1-925a-885af3c70de9","Type":"ContainerDied","Data":"ee3e5a11101cd896236cb369f1a0ae471184e838d73a92c5e7509279040b701f"}
Jul 09 18:06:44.044749 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.044585    2560 scope.go:117] "RemoveContainer" containerID="c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d"
Jul 09 18:06:44.056338 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.056316    2560 scope.go:117] "RemoveContainer" containerID="7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d"
Jul 09 18:06:44.063541 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.063521    2560 scope.go:117] "RemoveContainer" containerID="72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d"
Jul 09 18:06:44.070708 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.070683    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:44.072130 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.072106    2560 scope.go:117] "RemoveContainer" containerID="0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702"
Jul 09 18:06:44.073875 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.073853    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:44.078936 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.078914    2560 scope.go:117] "RemoveContainer" containerID="8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0"
Jul 09 18:06:44.085862 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.085845    2560 scope.go:117] "RemoveContainer" containerID="cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d"
Jul 09 18:06:44.093994 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.093977    2560 scope.go:117] "RemoveContainer" containerID="9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed"
Jul 09 18:06:44.101040 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101001    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:44.101275 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101263    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="alertmanager"
Jul 09 18:06:44.101314 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101276    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="alertmanager"
Jul 09 18:06:44.101314 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101289    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="init-config-reloader"
Jul 09 18:06:44.101314 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101295    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="init-config-reloader"
Jul 09 18:06:44.101314 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101303    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy"
Jul 09 18:06:44.101314 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101310    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101321    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="config-reloader"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101327    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="config-reloader"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101334    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy-metric"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101339    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy-metric"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101345    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="prom-label-proxy"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101351    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="prom-label-proxy"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101359    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy-web"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101363    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy-web"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101404    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="alertmanager"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101411    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101417    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="config-reloader"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101423    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy-metric"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101428    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="kube-rbac-proxy-web"
Jul 09 18:06:44.101515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.101433    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" containerName="prom-label-proxy"
Jul 09 18:06:44.102282 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.102262    2560 scope.go:117] "RemoveContainer" containerID="c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d"
Jul 09 18:06:44.102569 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:06:44.102542    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d\": container with ID starting with c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d not found: ID does not exist" containerID="c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d"
Jul 09 18:06:44.102656 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.102576    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d"} err="failed to get container status \"c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d\": rpc error: code = NotFound desc = could not find container \"c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d\": container with ID starting with c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d not found: ID does not exist"
Jul 09 18:06:44.102656 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.102594    2560 scope.go:117] "RemoveContainer" containerID="7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d"
Jul 09 18:06:44.102891 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:06:44.102868    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d\": container with ID starting with 7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d not found: ID does not exist" containerID="7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d"
Jul 09 18:06:44.102953 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.102899    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d"} err="failed to get container status \"7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d\": rpc error: code = NotFound desc = could not find container \"7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d\": container with ID starting with 7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d not found: ID does not exist"
Jul 09 18:06:44.102953 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.102916    2560 scope.go:117] "RemoveContainer" containerID="72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d"
Jul 09 18:06:44.103207 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:06:44.103188    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d\": container with ID starting with 72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d not found: ID does not exist" containerID="72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d"
Jul 09 18:06:44.103260 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.103214    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d"} err="failed to get container status \"72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d\": rpc error: code = NotFound desc = could not find container \"72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d\": container with ID starting with 72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d not found: ID does not exist"
Jul 09 18:06:44.103260 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.103233    2560 scope.go:117] "RemoveContainer" containerID="0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702"
Jul 09 18:06:44.103519 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:06:44.103501    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702\": container with ID starting with 0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702 not found: ID does not exist" containerID="0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702"
Jul 09 18:06:44.103591 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.103525    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702"} err="failed to get container status \"0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702\": rpc error: code = NotFound desc = could not find container \"0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702\": container with ID starting with 0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702 not found: ID does not exist"
Jul 09 18:06:44.103591 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.103558    2560 scope.go:117] "RemoveContainer" containerID="8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0"
Jul 09 18:06:44.103791 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:06:44.103777    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0\": container with ID starting with 8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0 not found: ID does not exist" containerID="8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0"
Jul 09 18:06:44.103836 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.103792    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0"} err="failed to get container status \"8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0\": rpc error: code = NotFound desc = could not find container \"8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0\": container with ID starting with 8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0 not found: ID does not exist"
Jul 09 18:06:44.103836 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.103802    2560 scope.go:117] "RemoveContainer" containerID="cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d"
Jul 09 18:06:44.104016 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:06:44.104002    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d\": container with ID starting with cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d not found: ID does not exist" containerID="cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d"
Jul 09 18:06:44.104063 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.104021    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d"} err="failed to get container status \"cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d\": rpc error: code = NotFound desc = could not find container \"cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d\": container with ID starting with cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d not found: ID does not exist"
Jul 09 18:06:44.104063 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.104032    2560 scope.go:117] "RemoveContainer" containerID="9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed"
Jul 09 18:06:44.104269 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:06:44.104252    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed\": container with ID starting with 9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed not found: ID does not exist" containerID="9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed"
Jul 09 18:06:44.104306 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.104274    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed"} err="failed to get container status \"9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed\": rpc error: code = NotFound desc = could not find container \"9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed\": container with ID starting with 9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed not found: ID does not exist"
Jul 09 18:06:44.104306 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.104290    2560 scope.go:117] "RemoveContainer" containerID="c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d"
Jul 09 18:06:44.104521 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.104504    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d"} err="failed to get container status \"c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d\": rpc error: code = NotFound desc = could not find container \"c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d\": container with ID starting with c242707558de2a6bcbbe7968d00a71ea2c545a089e9dd8403575078d70ddfc3d not found: ID does not exist"
Jul 09 18:06:44.104578 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.104521    2560 scope.go:117] "RemoveContainer" containerID="7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d"
Jul 09 18:06:44.104726 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.104711    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d"} err="failed to get container status \"7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d\": rpc error: code = NotFound desc = could not find container \"7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d\": container with ID starting with 7672d23dc1af93de37447878d95409ef78466fd0c37265021f6dce1d0c90235d not found: ID does not exist"
Jul 09 18:06:44.104771 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.104725    2560 scope.go:117] "RemoveContainer" containerID="72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d"
Jul 09 18:06:44.104771 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.104738    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.105143 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.105116    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d"} err="failed to get container status \"72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d\": rpc error: code = NotFound desc = could not find container \"72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d\": container with ID starting with 72192f7755145dbddead906aab213b92b1e8ae4665d6d04cb62abd0864f5402d not found: ID does not exist"
Jul 09 18:06:44.105212 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.105143    2560 scope.go:117] "RemoveContainer" containerID="0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702"
Jul 09 18:06:44.105423 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.105400    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702"} err="failed to get container status \"0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702\": rpc error: code = NotFound desc = could not find container \"0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702\": container with ID starting with 0395ca86f4ab81a37384dd7856236a3b107d4fa0ae455376345b6a94fcbb8702 not found: ID does not exist"
Jul 09 18:06:44.105509 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.105424    2560 scope.go:117] "RemoveContainer" containerID="8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0"
Jul 09 18:06:44.105681 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.105663    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0"} err="failed to get container status \"8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0\": rpc error: code = NotFound desc = could not find container \"8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0\": container with ID starting with 8536a38ce8fb9356fe1585c7f92ebf123b8194d19e034d64ddbabb170fc8bbe0 not found: ID does not exist"
Jul 09 18:06:44.105736 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.105682    2560 scope.go:117] "RemoveContainer" containerID="cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d"
Jul 09 18:06:44.105888 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.105869    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d"} err="failed to get container status \"cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d\": rpc error: code = NotFound desc = could not find container \"cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d\": container with ID starting with cd5c453ec675e69a339f16f096fab24946ed800a2d8bd12169265f1b9a38764d not found: ID does not exist"
Jul 09 18:06:44.105967 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.105890    2560 scope.go:117] "RemoveContainer" containerID="9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed"
Jul 09 18:06:44.106077 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.106059    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed"} err="failed to get container status \"9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed\": rpc error: code = NotFound desc = could not find container \"9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed\": container with ID starting with 9dbcec33af6e681ae6c72aa05445fb075729fc47b5eef12c74170e6fdf0027ed not found: ID does not exist"
Jul 09 18:06:44.107577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.107556    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 09 18:06:44.107577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.107570    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 09 18:06:44.107713 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.107580    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 09 18:06:44.107713 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.107594    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 09 18:06:44.107713 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.107594    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 09 18:06:44.107916 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.107902    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 09 18:06:44.108186 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.108170    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 09 18:06:44.108257 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.108241    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 09 18:06:44.108385 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.108369    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-w6gq4\""
Jul 09 18:06:44.113093 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.113073    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 09 18:06:44.120685 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.120665    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:44.221610 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221695 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221695 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-config-out\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221695 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221656    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-web-config\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221695 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221675    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221817 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221817 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221787    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-config-volume\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221817 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc259\" (UniqueName: \"kubernetes.io/projected/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-kube-api-access-qc259\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221902 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221902 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221849    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221902 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221984 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.221984 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.221960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322427 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322427 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322403    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-config-volume\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322427 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc259\" (UniqueName: \"kubernetes.io/projected/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-kube-api-access-qc259\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322651 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322651 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322651 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322651 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322873 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322873 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.322873 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.323015 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-config-out\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.323015 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-web-config\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.323015 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.323015 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.322987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.323602 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.323572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.324100 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.324042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.325952 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.325911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-config-volume\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.326055 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.326035    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.326110 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.326070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.326192 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.326104    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.326394 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.326372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-config-out\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.326625 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.326601    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.327009 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.326983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-web-config\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.327110 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.327076    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.327473 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.327453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.331011 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.330985    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc259\" (UniqueName: \"kubernetes.io/projected/e6c9aafc-8bcb-4ee8-a984-663d076dba1e-kube-api-access-qc259\") pod \"alertmanager-main-0\" (UID: \"e6c9aafc-8bcb-4ee8-a984-663d076dba1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.387967 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.387941    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7cec2df-4e4f-4fa1-925a-885af3c70de9" path="/var/lib/kubelet/pods/f7cec2df-4e4f-4fa1-925a-885af3c70de9/volumes"
Jul 09 18:06:44.413551 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.413534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.542149 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:44.542119    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:44.542278 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:06:44.542254    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c9aafc_8bcb_4ee8_a984_663d076dba1e.slice/crio-34b7486d9a5803366be6e40c63e40861afea16713dba534dd2372bc853af66cc WatchSource:0}: Error finding container 34b7486d9a5803366be6e40c63e40861afea16713dba534dd2372bc853af66cc: Status 404 returned error can't find the container with id 34b7486d9a5803366be6e40c63e40861afea16713dba534dd2372bc853af66cc
Jul 09 18:06:45.050338 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:45.050298    2560 generic.go:356] "Generic (PLEG): container finished" podID="e6c9aafc-8bcb-4ee8-a984-663d076dba1e" containerID="1ca5ea946dc2ebfb58410895e797a0ccbadc2310ef5962a43fb81594ade0df75" exitCode=0
Jul 09 18:06:45.050735 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:45.050387    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6c9aafc-8bcb-4ee8-a984-663d076dba1e","Type":"ContainerDied","Data":"1ca5ea946dc2ebfb58410895e797a0ccbadc2310ef5962a43fb81594ade0df75"}
Jul 09 18:06:45.050735 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:45.050470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6c9aafc-8bcb-4ee8-a984-663d076dba1e","Type":"ContainerStarted","Data":"34b7486d9a5803366be6e40c63e40861afea16713dba534dd2372bc853af66cc"}
Jul 09 18:06:46.058850 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:46.058807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6c9aafc-8bcb-4ee8-a984-663d076dba1e","Type":"ContainerStarted","Data":"3a94c915a82aeb5f50608f69e97dd9362ee2c6b47989cd67be647433eca8aa9d"}
Jul 09 18:06:46.059174 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:46.058917    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6c9aafc-8bcb-4ee8-a984-663d076dba1e","Type":"ContainerStarted","Data":"c475cb2bfae2226e057d097046ea0638f61a0f2930653ba4dd3c1350cfbc692b"}
Jul 09 18:06:46.059174 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:46.058956    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6c9aafc-8bcb-4ee8-a984-663d076dba1e","Type":"ContainerStarted","Data":"3dbfd90055dafa1021d4e1ea643df179a070282a20409e991be149dd8e4f9ecd"}
Jul 09 18:06:46.059174 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:46.058983    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6c9aafc-8bcb-4ee8-a984-663d076dba1e","Type":"ContainerStarted","Data":"5e42b95582055347420e1c03afb0df5ad0fe2754a9cce049a91acff93ca53be0"}
Jul 09 18:06:46.059174 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:46.059005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6c9aafc-8bcb-4ee8-a984-663d076dba1e","Type":"ContainerStarted","Data":"794e4fb2aeab5b5619264c424abac6e425d9fccc7f8272c0ca1da3b38872d84b"}
Jul 09 18:06:46.059174 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:46.059052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6c9aafc-8bcb-4ee8-a984-663d076dba1e","Type":"ContainerStarted","Data":"1c65cfac77c782dea7b5418e379fe450f5df2f49094ed9b2e0068532a4eda172"}
Jul 09 18:06:46.087330 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:46.087287    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.071782337 podStartE2EDuration="2.0872729s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.490563ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:46.086439994 +0000 UTC m=+150.238077291" watchObservedRunningTime="2026-07-09 18:06:46.0872729 +0000 UTC m=+150.238910172"
Jul 09 18:06:49.540132 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:49.540097    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:49.540132 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:49.540147    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:49.544579 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:49.544561    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:49.791412 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:49.791346    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:49.897145 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:49.897121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:50.077192 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:50.077146    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:06:50.128348 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:50.128326    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845b6dbd9d-c57dg"]
Jul 09 18:06:50.239819 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:50.239796    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:50.301893 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:50.301865    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:50.371154 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:06:50.371082    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:02.179702 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:02.179656    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:02.280425 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:02.280378    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:08.660100 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:08.659999    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:15.148182 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.148118    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-845b6dbd9d-c57dg" podUID="2e9383d9-fd56-4987-833f-96f46c348eaa" containerName="console" containerID="cri-o://7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb" gracePeriod=15
Jul 09 18:07:15.394267 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.394244    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:07:15.545507 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.545466    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-service-ca\") pod \"2e9383d9-fd56-4987-833f-96f46c348eaa\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") "
Jul 09 18:07:15.545685 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.545542    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-trusted-ca-bundle\") pod \"2e9383d9-fd56-4987-833f-96f46c348eaa\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") "
Jul 09 18:07:15.545685 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.545595    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-oauth-config\") pod \"2e9383d9-fd56-4987-833f-96f46c348eaa\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") "
Jul 09 18:07:15.545685 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.545614    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-serving-cert\") pod \"2e9383d9-fd56-4987-833f-96f46c348eaa\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") "
Jul 09 18:07:15.545685 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.545641    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-oauth-serving-cert\") pod \"2e9383d9-fd56-4987-833f-96f46c348eaa\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") "
Jul 09 18:07:15.545685 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.545664    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-console-config\") pod \"2e9383d9-fd56-4987-833f-96f46c348eaa\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") "
Jul 09 18:07:15.545948 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.545719    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwnx5\" (UniqueName: \"kubernetes.io/projected/2e9383d9-fd56-4987-833f-96f46c348eaa-kube-api-access-hwnx5\") pod \"2e9383d9-fd56-4987-833f-96f46c348eaa\" (UID: \"2e9383d9-fd56-4987-833f-96f46c348eaa\") "
Jul 09 18:07:15.545948 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.545796    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-service-ca" (OuterVolumeSpecName: "service-ca") pod "2e9383d9-fd56-4987-833f-96f46c348eaa" (UID: "2e9383d9-fd56-4987-833f-96f46c348eaa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:15.546039 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.546008    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2e9383d9-fd56-4987-833f-96f46c348eaa" (UID: "2e9383d9-fd56-4987-833f-96f46c348eaa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:15.546039 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.546019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2e9383d9-fd56-4987-833f-96f46c348eaa" (UID: "2e9383d9-fd56-4987-833f-96f46c348eaa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:15.546114 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.546060    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-console-config" (OuterVolumeSpecName: "console-config") pod "2e9383d9-fd56-4987-833f-96f46c348eaa" (UID: "2e9383d9-fd56-4987-833f-96f46c348eaa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:15.547649 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.547616    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2e9383d9-fd56-4987-833f-96f46c348eaa" (UID: "2e9383d9-fd56-4987-833f-96f46c348eaa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:07:15.547753 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.547691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2e9383d9-fd56-4987-833f-96f46c348eaa" (UID: "2e9383d9-fd56-4987-833f-96f46c348eaa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:07:15.547753 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.547720    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9383d9-fd56-4987-833f-96f46c348eaa-kube-api-access-hwnx5" (OuterVolumeSpecName: "kube-api-access-hwnx5") pod "2e9383d9-fd56-4987-833f-96f46c348eaa" (UID: "2e9383d9-fd56-4987-833f-96f46c348eaa"). InnerVolumeSpecName "kube-api-access-hwnx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:07:15.646395 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.646363    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-oauth-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.646395 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.646388    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e9383d9-fd56-4987-833f-96f46c348eaa-console-serving-cert\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.646395 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.646398    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-oauth-serving-cert\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.646630 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.646407    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-console-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.646630 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.646416    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwnx5\" (UniqueName: \"kubernetes.io/projected/2e9383d9-fd56-4987-833f-96f46c348eaa-kube-api-access-hwnx5\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.646630 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.646425    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-service-ca\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.646630 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:15.646433    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e9383d9-fd56-4987-833f-96f46c348eaa-trusted-ca-bundle\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:16.172734 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.172699    2560 generic.go:356] "Generic (PLEG): container finished" podID="2e9383d9-fd56-4987-833f-96f46c348eaa" containerID="7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb" exitCode=0
Jul 09 18:07:16.173137 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.172759    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845b6dbd9d-c57dg"
Jul 09 18:07:16.173137 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.172783    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845b6dbd9d-c57dg" event={"ID":"2e9383d9-fd56-4987-833f-96f46c348eaa","Type":"ContainerDied","Data":"7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb"}
Jul 09 18:07:16.173137 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.172860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845b6dbd9d-c57dg" event={"ID":"2e9383d9-fd56-4987-833f-96f46c348eaa","Type":"ContainerDied","Data":"6cdded58b7e4fd02f7c7f7ae6ae1e67e4ec7cb41dac778bcf373468d72ed9ba2"}
Jul 09 18:07:16.173137 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.172881    2560 scope.go:117] "RemoveContainer" containerID="7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb"
Jul 09 18:07:16.181346 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.181331    2560 scope.go:117] "RemoveContainer" containerID="7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb"
Jul 09 18:07:16.181607 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:07:16.181590    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb\": container with ID starting with 7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb not found: ID does not exist" containerID="7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb"
Jul 09 18:07:16.181655 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.181614    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb"} err="failed to get container status \"7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb\": rpc error: code = NotFound desc = could not find container \"7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb\": container with ID starting with 7f4e5b1d6ed1089789b514f00544e3e8ed833aab063680067f4ea9c749f15bfb not found: ID does not exist"
Jul 09 18:07:16.193349 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.193328    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845b6dbd9d-c57dg"]
Jul 09 18:07:16.197291 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.197274    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-845b6dbd9d-c57dg"]
Jul 09 18:07:16.387700 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:16.387673    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e9383d9-fd56-4987-833f-96f46c348eaa" path="/var/lib/kubelet/pods/2e9383d9-fd56-4987-833f-96f46c348eaa/volumes"
Jul 09 18:07:28.014161 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:28.014121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:40.912136 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:40.912091    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:41.012104 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:41.012072    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:52.952543 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:07:52.952474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:34.286097 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:34.286051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:34.385283 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:34.385253    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:34.648777 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:34.648698    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:34.745976 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:34.745948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:34.820082 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:34.820055    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:35.991667 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:35.991629    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"]
Jul 09 18:08:35.992031 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:35.991919    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9383d9-fd56-4987-833f-96f46c348eaa" containerName="console"
Jul 09 18:08:35.992031 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:35.991930    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9383d9-fd56-4987-833f-96f46c348eaa" containerName="console"
Jul 09 18:08:35.992031 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:35.991975    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9383d9-fd56-4987-833f-96f46c348eaa" containerName="console"
Jul 09 18:08:35.993897 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:35.993877    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:35.998174 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:35.998154    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n2ktc\""
Jul 09 18:08:35.998256 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:35.998177    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 18:08:35.999150 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:35.999134    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 18:08:36.005255 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.005231    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"]
Jul 09 18:08:36.112533 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.112509    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.112626 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.112556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.112626 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.112591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kkv\" (UniqueName: \"kubernetes.io/projected/b2a1d981-5083-43de-87fb-9358588042c5-kube-api-access-m7kkv\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.213439 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.213396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.213577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.213504    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kkv\" (UniqueName: \"kubernetes.io/projected/b2a1d981-5083-43de-87fb-9358588042c5-kube-api-access-m7kkv\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.213577 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.213556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.213802 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.213780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.213947 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.213928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.226993 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.226958    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kkv\" (UniqueName: \"kubernetes.io/projected/b2a1d981-5083-43de-87fb-9358588042c5-kube-api-access-m7kkv\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.303451 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.303386    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:36.435295 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.435192    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"]
Jul 09 18:08:36.437760 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:08:36.437735    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a1d981_5083_43de_87fb_9358588042c5.slice/crio-6dec1e005e02de7a3dec5c8a26bd103f9fd19a109093d4da49430ee2f6f8d674 WatchSource:0}: Error finding container 6dec1e005e02de7a3dec5c8a26bd103f9fd19a109093d4da49430ee2f6f8d674: Status 404 returned error can't find the container with id 6dec1e005e02de7a3dec5c8a26bd103f9fd19a109093d4da49430ee2f6f8d674
Jul 09 18:08:36.465709 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:36.465682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26" event={"ID":"b2a1d981-5083-43de-87fb-9358588042c5","Type":"ContainerStarted","Data":"6dec1e005e02de7a3dec5c8a26bd103f9fd19a109093d4da49430ee2f6f8d674"}
Jul 09 18:08:41.487629 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:41.487590    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26" event={"ID":"b2a1d981-5083-43de-87fb-9358588042c5","Type":"ContainerStarted","Data":"4bc8c815096fc042dcbddc8a89fe4cec7411f0ea2839fc6edab8c7b0b0d5a8e7"}
Jul 09 18:08:42.492663 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:42.492618    2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a1d981-5083-43de-87fb-9358588042c5" containerID="4bc8c815096fc042dcbddc8a89fe4cec7411f0ea2839fc6edab8c7b0b0d5a8e7" exitCode=0
Jul 09 18:08:42.493094 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:42.492703    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26" event={"ID":"b2a1d981-5083-43de-87fb-9358588042c5","Type":"ContainerDied","Data":"4bc8c815096fc042dcbddc8a89fe4cec7411f0ea2839fc6edab8c7b0b0d5a8e7"}
Jul 09 18:08:44.502145 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:44.502113    2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a1d981-5083-43de-87fb-9358588042c5" containerID="241595b23757565fb8e596bb503e503f854d0eeedc1102234de97a32a411cd49" exitCode=0
Jul 09 18:08:44.502425 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:44.502187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26" event={"ID":"b2a1d981-5083-43de-87fb-9358588042c5","Type":"ContainerDied","Data":"241595b23757565fb8e596bb503e503f854d0eeedc1102234de97a32a411cd49"}
Jul 09 18:08:50.528787 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:50.528753    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26" event={"ID":"b2a1d981-5083-43de-87fb-9358588042c5","Type":"ContainerStarted","Data":"6f59e45967f07724e5dee8373a329ab8a2e47c0d41c439392d0f01498b62a0f0"}
Jul 09 18:08:50.551145 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:50.550810    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26" podStartSLOduration=2.967016548 podStartE2EDuration="15.550790367s" totalImagesPullingTime="12.583773819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:50.548742331 +0000 UTC m=+274.700379627" watchObservedRunningTime="2026-07-09 18:08:50.550790367 +0000 UTC m=+274.702427640"
Jul 09 18:08:51.535043 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:51.535004    2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a1d981-5083-43de-87fb-9358588042c5" containerID="6f59e45967f07724e5dee8373a329ab8a2e47c0d41c439392d0f01498b62a0f0" exitCode=0
Jul 09 18:08:51.535441 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:51.535054    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26" event={"ID":"b2a1d981-5083-43de-87fb-9358588042c5","Type":"ContainerDied","Data":"6f59e45967f07724e5dee8373a329ab8a2e47c0d41c439392d0f01498b62a0f0"}
Jul 09 18:08:52.668949 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.668927    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:52.747656 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.747632    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-util\") pod \"b2a1d981-5083-43de-87fb-9358588042c5\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") "
Jul 09 18:08:52.747758 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.747669    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-bundle\") pod \"b2a1d981-5083-43de-87fb-9358588042c5\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") "
Jul 09 18:08:52.747758 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.747739    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7kkv\" (UniqueName: \"kubernetes.io/projected/b2a1d981-5083-43de-87fb-9358588042c5-kube-api-access-m7kkv\") pod \"b2a1d981-5083-43de-87fb-9358588042c5\" (UID: \"b2a1d981-5083-43de-87fb-9358588042c5\") "
Jul 09 18:08:52.748283 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.748260    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-bundle" (OuterVolumeSpecName: "bundle") pod "b2a1d981-5083-43de-87fb-9358588042c5" (UID: "b2a1d981-5083-43de-87fb-9358588042c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:08:52.749766 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.749733    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a1d981-5083-43de-87fb-9358588042c5-kube-api-access-m7kkv" (OuterVolumeSpecName: "kube-api-access-m7kkv") pod "b2a1d981-5083-43de-87fb-9358588042c5" (UID: "b2a1d981-5083-43de-87fb-9358588042c5"). InnerVolumeSpecName "kube-api-access-m7kkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:08:52.751727 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.751706    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-util" (OuterVolumeSpecName: "util") pod "b2a1d981-5083-43de-87fb-9358588042c5" (UID: "b2a1d981-5083-43de-87fb-9358588042c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:08:52.848924 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.848868    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7kkv\" (UniqueName: \"kubernetes.io/projected/b2a1d981-5083-43de-87fb-9358588042c5-kube-api-access-m7kkv\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:08:52.848924 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.848896    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-util\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:08:52.848924 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:52.848906    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a1d981-5083-43de-87fb-9358588042c5-bundle\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:08:53.544461 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:53.544437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26" event={"ID":"b2a1d981-5083-43de-87fb-9358588042c5","Type":"ContainerDied","Data":"6dec1e005e02de7a3dec5c8a26bd103f9fd19a109093d4da49430ee2f6f8d674"}
Jul 09 18:08:53.544570 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:53.544498    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dec1e005e02de7a3dec5c8a26bd103f9fd19a109093d4da49430ee2f6f8d674"
Jul 09 18:08:53.544570 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:53.544501    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589c8t26"
Jul 09 18:08:57.860420 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:57.860389    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:58.032692 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:08:58.032657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:00.679416 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.679383    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"]
Jul 09 18:09:00.679875 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.679656    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a1d981-5083-43de-87fb-9358588042c5" containerName="util"
Jul 09 18:09:00.679875 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.679670    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a1d981-5083-43de-87fb-9358588042c5" containerName="util"
Jul 09 18:09:00.679875 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.679685    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a1d981-5083-43de-87fb-9358588042c5" containerName="extract"
Jul 09 18:09:00.679875 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.679691    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a1d981-5083-43de-87fb-9358588042c5" containerName="extract"
Jul 09 18:09:00.679875 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.679708    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a1d981-5083-43de-87fb-9358588042c5" containerName="pull"
Jul 09 18:09:00.679875 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.679714    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a1d981-5083-43de-87fb-9358588042c5" containerName="pull"
Jul 09 18:09:00.679875 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.679760    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a1d981-5083-43de-87fb-9358588042c5" containerName="extract"
Jul 09 18:09:00.708743 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.708716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:00.710232 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.710206    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"]
Jul 09 18:09:00.715108 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.715090    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 09 18:09:00.715508 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.715492    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 09 18:09:00.715626 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.715547    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 09 18:09:00.715899 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.715880    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-4fgrb\""
Jul 09 18:09:00.807433 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.807414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjb75\" (UniqueName: \"kubernetes.io/projected/40150a67-7aa1-40e7-bb25-46368cc7faa3-kube-api-access-fjb75\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2\" (UID: \"40150a67-7aa1-40e7-bb25-46368cc7faa3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:00.807548 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.807452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/40150a67-7aa1-40e7-bb25-46368cc7faa3-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2\" (UID: \"40150a67-7aa1-40e7-bb25-46368cc7faa3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:00.908211 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.908188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjb75\" (UniqueName: \"kubernetes.io/projected/40150a67-7aa1-40e7-bb25-46368cc7faa3-kube-api-access-fjb75\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2\" (UID: \"40150a67-7aa1-40e7-bb25-46368cc7faa3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:00.908308 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.908229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/40150a67-7aa1-40e7-bb25-46368cc7faa3-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2\" (UID: \"40150a67-7aa1-40e7-bb25-46368cc7faa3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:00.910541 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.910521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/40150a67-7aa1-40e7-bb25-46368cc7faa3-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2\" (UID: \"40150a67-7aa1-40e7-bb25-46368cc7faa3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:00.927537 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:00.927517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjb75\" (UniqueName: \"kubernetes.io/projected/40150a67-7aa1-40e7-bb25-46368cc7faa3-kube-api-access-fjb75\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2\" (UID: \"40150a67-7aa1-40e7-bb25-46368cc7faa3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:01.020997 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:01.020976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:01.183417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:01.183390    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"]
Jul 09 18:09:01.186889 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:09:01.186867    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40150a67_7aa1_40e7_bb25_46368cc7faa3.slice/crio-16f1ce6f86dda46407ea6506dc089ec0d5f8be90ed52b5b1d2ee3140d9d57b77 WatchSource:0}: Error finding container 16f1ce6f86dda46407ea6506dc089ec0d5f8be90ed52b5b1d2ee3140d9d57b77: Status 404 returned error can't find the container with id 16f1ce6f86dda46407ea6506dc089ec0d5f8be90ed52b5b1d2ee3140d9d57b77
Jul 09 18:09:01.574255 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:01.574223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2" event={"ID":"40150a67-7aa1-40e7-bb25-46368cc7faa3","Type":"ContainerStarted","Data":"16f1ce6f86dda46407ea6506dc089ec0d5f8be90ed52b5b1d2ee3140d9d57b77"}
Jul 09 18:09:07.032624 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.032591    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:07.391821 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.391736    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:07.400475 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.400441    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-dggps"]
Jul 09 18:09:07.402414 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.402395    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:07.405980 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.405961    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 09 18:09:07.406054 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.405979    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6dhsf\""
Jul 09 18:09:07.406054 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.405986    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 09 18:09:07.426616 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.426595    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-dggps"]
Jul 09 18:09:07.554079 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.554054    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-certificates\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:07.554198 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.554107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a0160260-444c-4f3b-8532-3a9d62f4103c-cabundle0\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:07.554247 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.554202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq66m\" (UniqueName: \"kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-kube-api-access-sq66m\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:07.598505 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.598458    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2" event={"ID":"40150a67-7aa1-40e7-bb25-46368cc7faa3","Type":"ContainerStarted","Data":"cc53e1fc37e5d4b4229e5a7be7969f88bac491241c3331aded18566d6361344d"}
Jul 09 18:09:07.598608 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.598568    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:07.626892 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.626854    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2" podStartSLOduration=2.216196327 podStartE2EDuration="7.626841227s" totalImagesPullingTime="5.4106449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:07.625622499 +0000 UTC m=+291.777259770" watchObservedRunningTime="2026-07-09 18:09:07.626841227 +0000 UTC m=+291.778478499"
Jul 09 18:09:07.654919 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.654872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-certificates\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:07.654919 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.654911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a0160260-444c-4f3b-8532-3a9d62f4103c-cabundle0\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:07.655039 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.654943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq66m\" (UniqueName: \"kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-kube-api-access-sq66m\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:07.655150 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:07.655137    2560 secret.go:281] references non-existent secret key: ca.crt
Jul 09 18:09:07.655185 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:07.655153    2560 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 09 18:09:07.655185 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:07.655161    2560 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-dggps: references non-existent secret key: ca.crt
Jul 09 18:09:07.655248 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:07.655204    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-certificates podName:a0160260-444c-4f3b-8532-3a9d62f4103c nodeName:}" failed. No retries permitted until 2026-07-09 18:09:08.155190635 +0000 UTC m=+292.306827884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-certificates") pod "keda-operator-68bff59c-dggps" (UID: "a0160260-444c-4f3b-8532-3a9d62f4103c") : references non-existent secret key: ca.crt
Jul 09 18:09:07.655507 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.655468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a0160260-444c-4f3b-8532-3a9d62f4103c-cabundle0\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:07.670317 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:07.670291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq66m\" (UniqueName: \"kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-kube-api-access-sq66m\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:08.158292 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.158268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-certificates\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:08.158719 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:08.158421    2560 secret.go:281] references non-existent secret key: ca.crt
Jul 09 18:09:08.158719 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:08.158443    2560 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 09 18:09:08.158719 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:08.158456    2560 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-dggps: references non-existent secret key: ca.crt
Jul 09 18:09:08.158719 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:08.158535    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-certificates podName:a0160260-444c-4f3b-8532-3a9d62f4103c nodeName:}" failed. No retries permitted until 2026-07-09 18:09:09.158516765 +0000 UTC m=+293.310154017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-certificates") pod "keda-operator-68bff59c-dggps" (UID: "a0160260-444c-4f3b-8532-3a9d62f4103c") : references non-existent secret key: ca.crt
Jul 09 18:09:08.208196 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.208172    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"]
Jul 09 18:09:08.210257 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.210240    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.215494 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.215462    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 09 18:09:08.227595 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.227570    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"]
Jul 09 18:09:08.360229 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.360203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e1852787-dd85-494a-a1fc-5ee7d7db1efe-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-l9btl\" (UID: \"e1852787-dd85-494a-a1fc-5ee7d7db1efe\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.360332 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.360252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmd7f\" (UniqueName: \"kubernetes.io/projected/e1852787-dd85-494a-a1fc-5ee7d7db1efe-kube-api-access-wmd7f\") pod \"keda-metrics-apiserver-559f75f947-l9btl\" (UID: \"e1852787-dd85-494a-a1fc-5ee7d7db1efe\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.360332 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.360321    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e1852787-dd85-494a-a1fc-5ee7d7db1efe-certificates\") pod \"keda-metrics-apiserver-559f75f947-l9btl\" (UID: \"e1852787-dd85-494a-a1fc-5ee7d7db1efe\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.461508 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.461447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e1852787-dd85-494a-a1fc-5ee7d7db1efe-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-l9btl\" (UID: \"e1852787-dd85-494a-a1fc-5ee7d7db1efe\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.461508 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.461499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmd7f\" (UniqueName: \"kubernetes.io/projected/e1852787-dd85-494a-a1fc-5ee7d7db1efe-kube-api-access-wmd7f\") pod \"keda-metrics-apiserver-559f75f947-l9btl\" (UID: \"e1852787-dd85-494a-a1fc-5ee7d7db1efe\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.461621 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.461546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e1852787-dd85-494a-a1fc-5ee7d7db1efe-certificates\") pod \"keda-metrics-apiserver-559f75f947-l9btl\" (UID: \"e1852787-dd85-494a-a1fc-5ee7d7db1efe\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.461855 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.461837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e1852787-dd85-494a-a1fc-5ee7d7db1efe-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-l9btl\" (UID: \"e1852787-dd85-494a-a1fc-5ee7d7db1efe\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.463919 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.463893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e1852787-dd85-494a-a1fc-5ee7d7db1efe-certificates\") pod \"keda-metrics-apiserver-559f75f947-l9btl\" (UID: \"e1852787-dd85-494a-a1fc-5ee7d7db1efe\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.474855 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.474835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmd7f\" (UniqueName: \"kubernetes.io/projected/e1852787-dd85-494a-a1fc-5ee7d7db1efe-kube-api-access-wmd7f\") pod \"keda-metrics-apiserver-559f75f947-l9btl\" (UID: \"e1852787-dd85-494a-a1fc-5ee7d7db1efe\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.520652 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.520633    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:08.591041 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.591010    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-l2r5k"]
Jul 09 18:09:08.593729 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.593711    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:08.597396 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.597373    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 09 18:09:08.608651 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.608606    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-l2r5k"]
Jul 09 18:09:08.647765 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.647735    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:08.654810 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.654789    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"]
Jul 09 18:09:08.657806 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:09:08.657778    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1852787_dd85_494a_a1fc_5ee7d7db1efe.slice/crio-ac642a53aef9cbda8959bb0522269e7819a92c5a92d72e61468573e22386f236 WatchSource:0}: Error finding container ac642a53aef9cbda8959bb0522269e7819a92c5a92d72e61468573e22386f236: Status 404 returned error can't find the container with id ac642a53aef9cbda8959bb0522269e7819a92c5a92d72e61468573e22386f236
Jul 09 18:09:08.765095 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.765071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zntw\" (UniqueName: \"kubernetes.io/projected/57167af0-a132-4b71-be0f-36adcfd056ca-kube-api-access-6zntw\") pod \"keda-admission-5fbd5c4644-l2r5k\" (UID: \"57167af0-a132-4b71-be0f-36adcfd056ca\") " pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:08.765178 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.765123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/57167af0-a132-4b71-be0f-36adcfd056ca-certificates\") pod \"keda-admission-5fbd5c4644-l2r5k\" (UID: \"57167af0-a132-4b71-be0f-36adcfd056ca\") " pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:08.866470 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.866448    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zntw\" (UniqueName: \"kubernetes.io/projected/57167af0-a132-4b71-be0f-36adcfd056ca-kube-api-access-6zntw\") pod \"keda-admission-5fbd5c4644-l2r5k\" (UID: \"57167af0-a132-4b71-be0f-36adcfd056ca\") " pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:08.866569 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.866499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/57167af0-a132-4b71-be0f-36adcfd056ca-certificates\") pod \"keda-admission-5fbd5c4644-l2r5k\" (UID: \"57167af0-a132-4b71-be0f-36adcfd056ca\") " pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:08.866611 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:08.866584    2560 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 09 18:09:08.866611 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:08.866599    2560 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-l2r5k: secret "keda-admission-webhooks-certs" not found
Jul 09 18:09:08.866672 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:09:08.866648    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57167af0-a132-4b71-be0f-36adcfd056ca-certificates podName:57167af0-a132-4b71-be0f-36adcfd056ca nodeName:}" failed. No retries permitted until 2026-07-09 18:09:09.366635687 +0000 UTC m=+293.518272940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/57167af0-a132-4b71-be0f-36adcfd056ca-certificates") pod "keda-admission-5fbd5c4644-l2r5k" (UID: "57167af0-a132-4b71-be0f-36adcfd056ca") : secret "keda-admission-webhooks-certs" not found
Jul 09 18:09:08.877101 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.877079    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zntw\" (UniqueName: \"kubernetes.io/projected/57167af0-a132-4b71-be0f-36adcfd056ca-kube-api-access-6zntw\") pod \"keda-admission-5fbd5c4644-l2r5k\" (UID: \"57167af0-a132-4b71-be0f-36adcfd056ca\") " pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:08.993762 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:08.993739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:09.078246 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.078190    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:09.168798 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.168776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-certificates\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:09.175073 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.175050    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a0160260-444c-4f3b-8532-3a9d62f4103c-certificates\") pod \"keda-operator-68bff59c-dggps\" (UID: \"a0160260-444c-4f3b-8532-3a9d62f4103c\") " pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:09.184875 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.184851    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:09.211789 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.211766    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:09.286596 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.286511    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:09.352186 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.352158    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-dggps"]
Jul 09 18:09:09.354143 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:09:09.354083    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0160260_444c_4f3b_8532_3a9d62f4103c.slice/crio-cedf567172e475539a9f50392cbd36dd82c3d17a848c26dc54c91eaf294c3384 WatchSource:0}: Error finding container cedf567172e475539a9f50392cbd36dd82c3d17a848c26dc54c91eaf294c3384: Status 404 returned error can't find the container with id cedf567172e475539a9f50392cbd36dd82c3d17a848c26dc54c91eaf294c3384
Jul 09 18:09:09.356177 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.356148    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:09.370545 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.370521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/57167af0-a132-4b71-be0f-36adcfd056ca-certificates\") pod \"keda-admission-5fbd5c4644-l2r5k\" (UID: \"57167af0-a132-4b71-be0f-36adcfd056ca\") " pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:09.372692 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.372673    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/57167af0-a132-4b71-be0f-36adcfd056ca-certificates\") pod \"keda-admission-5fbd5c4644-l2r5k\" (UID: \"57167af0-a132-4b71-be0f-36adcfd056ca\") " pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:09.510383 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.510357    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:09.615674 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.615589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-dggps" event={"ID":"a0160260-444c-4f3b-8532-3a9d62f4103c","Type":"ContainerStarted","Data":"cedf567172e475539a9f50392cbd36dd82c3d17a848c26dc54c91eaf294c3384"}
Jul 09 18:09:09.616910 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.616878    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl" event={"ID":"e1852787-dd85-494a-a1fc-5ee7d7db1efe","Type":"ContainerStarted","Data":"ac642a53aef9cbda8959bb0522269e7819a92c5a92d72e61468573e22386f236"}
Jul 09 18:09:09.653073 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:09.653041    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-l2r5k"]
Jul 09 18:09:09.655973 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:09:09.655950    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57167af0_a132_4b71_be0f_36adcfd056ca.slice/crio-96f1eb6284e2e22cc1ec60d7eccdba7f77cff706c1b1eaa0f3eb4dcece60a270 WatchSource:0}: Error finding container 96f1eb6284e2e22cc1ec60d7eccdba7f77cff706c1b1eaa0f3eb4dcece60a270: Status 404 returned error can't find the container with id 96f1eb6284e2e22cc1ec60d7eccdba7f77cff706c1b1eaa0f3eb4dcece60a270
Jul 09 18:09:10.623629 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:10.623592    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k" event={"ID":"57167af0-a132-4b71-be0f-36adcfd056ca","Type":"ContainerStarted","Data":"96f1eb6284e2e22cc1ec60d7eccdba7f77cff706c1b1eaa0f3eb4dcece60a270"}
Jul 09 18:09:12.633704 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:12.633669    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k" event={"ID":"57167af0-a132-4b71-be0f-36adcfd056ca","Type":"ContainerStarted","Data":"35ee6873928e0c5354c4699f6b7ecdd0643b93e3e3f2fd7d8aa867c4643e64fd"}
Jul 09 18:09:12.634170 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:12.633802    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:12.635077 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:12.635053    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl" event={"ID":"e1852787-dd85-494a-a1fc-5ee7d7db1efe","Type":"ContainerStarted","Data":"e66a10c4d51582fb009dbeee02f890e95f6051aed3c44a75ca28e36bdf829a9e"}
Jul 09 18:09:12.635168 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:12.635101    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:12.652247 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:12.652202    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k" podStartSLOduration=2.432324429 podStartE2EDuration="4.652190884s" totalImagesPullingTime="2.219866455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:12.651566662 +0000 UTC m=+296.803203934" watchObservedRunningTime="2026-07-09 18:09:12.652190884 +0000 UTC m=+296.803828153"
Jul 09 18:09:12.674101 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:12.674050    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl" podStartSLOduration=1.450336168 podStartE2EDuration="4.674038632s" totalImagesPullingTime="3.223702464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:12.672410813 +0000 UTC m=+296.824048083" watchObservedRunningTime="2026-07-09 18:09:12.674038632 +0000 UTC m=+296.825675902"
Jul 09 18:09:15.653718 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:15.653683    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-dggps" event={"ID":"a0160260-444c-4f3b-8532-3a9d62f4103c","Type":"ContainerStarted","Data":"9b370692f5834b214040a18c866854b2e907095e754bea967bf3d83554074b83"}
Jul 09 18:09:15.654063 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:15.653761    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:09:15.676699 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:15.676658    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-dggps" podStartSLOduration=3.060109983 podStartE2EDuration="8.676643046s" totalImagesPullingTime="5.616533063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:15.674992653 +0000 UTC m=+299.826629924" watchObservedRunningTime="2026-07-09 18:09:15.676643046 +0000 UTC m=+299.828280317"
Jul 09 18:09:16.223040 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:16.223012    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:09:16.226020 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:16.226000    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:09:16.230177 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:16.230157    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:09:23.648423 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:23.648392    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-l9btl"
Jul 09 18:09:28.609973 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:28.609941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-7fdn2"
Jul 09 18:09:29.525316 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:29.525260    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:29.623265 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:29.623237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:33.646989 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:33.646956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-l2r5k"
Jul 09 18:09:36.658109 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:09:36.658074    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-dggps"
Jul 09 18:10:22.264195 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.264100    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:22.321626 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.321589    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-57b57d8c6-b8wgk"]
Jul 09 18:10:22.324631 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.324613    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:22.327133 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.327108    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 09 18:10:22.327259 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.327241    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:10:22.327391 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.327372    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:10:22.328066 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.328053    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-d2l2h\""
Jul 09 18:10:22.331408 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.331239    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-xb5ts"]
Jul 09 18:10:22.334417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.334397    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:22.334544 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.334466    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57b57d8c6-b8wgk"]
Jul 09 18:10:22.337047 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.337027    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 09 18:10:22.337143 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.337077    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-czq7r\""
Jul 09 18:10:22.343421 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.343396    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-xb5ts"]
Jul 09 18:10:22.348682 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.348655    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:22.411618 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.411578    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:22.420618 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.420590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rc6\" (UniqueName: \"kubernetes.io/projected/01817576-2fb0-4596-a7ad-a940d5c458b3-kube-api-access-h5rc6\") pod \"llmisvc-controller-manager-654c557856-xb5ts\" (UID: \"01817576-2fb0-4596-a7ad-a940d5c458b3\") " pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:22.420731 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.420645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a70793ac-4242-4a65-83ec-92f83314d266-cert\") pod \"kserve-controller-manager-57b57d8c6-b8wgk\" (UID: \"a70793ac-4242-4a65-83ec-92f83314d266\") " pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:22.420731 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.420699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01817576-2fb0-4596-a7ad-a940d5c458b3-cert\") pod \"llmisvc-controller-manager-654c557856-xb5ts\" (UID: \"01817576-2fb0-4596-a7ad-a940d5c458b3\") " pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:22.420799 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.420744    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d5xf\" (UniqueName: \"kubernetes.io/projected/a70793ac-4242-4a65-83ec-92f83314d266-kube-api-access-9d5xf\") pod \"kserve-controller-manager-57b57d8c6-b8wgk\" (UID: \"a70793ac-4242-4a65-83ec-92f83314d266\") " pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:22.469111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.469075    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:22.521473 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.521390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rc6\" (UniqueName: \"kubernetes.io/projected/01817576-2fb0-4596-a7ad-a940d5c458b3-kube-api-access-h5rc6\") pod \"llmisvc-controller-manager-654c557856-xb5ts\" (UID: \"01817576-2fb0-4596-a7ad-a940d5c458b3\") " pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:22.521671 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.521642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a70793ac-4242-4a65-83ec-92f83314d266-cert\") pod \"kserve-controller-manager-57b57d8c6-b8wgk\" (UID: \"a70793ac-4242-4a65-83ec-92f83314d266\") " pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:22.521736 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.521693    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01817576-2fb0-4596-a7ad-a940d5c458b3-cert\") pod \"llmisvc-controller-manager-654c557856-xb5ts\" (UID: \"01817576-2fb0-4596-a7ad-a940d5c458b3\") " pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:22.521976 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.521943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d5xf\" (UniqueName: \"kubernetes.io/projected/a70793ac-4242-4a65-83ec-92f83314d266-kube-api-access-9d5xf\") pod \"kserve-controller-manager-57b57d8c6-b8wgk\" (UID: \"a70793ac-4242-4a65-83ec-92f83314d266\") " pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:22.524414 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.524384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01817576-2fb0-4596-a7ad-a940d5c458b3-cert\") pod \"llmisvc-controller-manager-654c557856-xb5ts\" (UID: \"01817576-2fb0-4596-a7ad-a940d5c458b3\") " pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:22.524549 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.524384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a70793ac-4242-4a65-83ec-92f83314d266-cert\") pod \"kserve-controller-manager-57b57d8c6-b8wgk\" (UID: \"a70793ac-4242-4a65-83ec-92f83314d266\") " pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:22.530337 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.530311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d5xf\" (UniqueName: \"kubernetes.io/projected/a70793ac-4242-4a65-83ec-92f83314d266-kube-api-access-9d5xf\") pod \"kserve-controller-manager-57b57d8c6-b8wgk\" (UID: \"a70793ac-4242-4a65-83ec-92f83314d266\") " pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:22.530542 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.530516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rc6\" (UniqueName: \"kubernetes.io/projected/01817576-2fb0-4596-a7ad-a940d5c458b3-kube-api-access-h5rc6\") pod \"llmisvc-controller-manager-654c557856-xb5ts\" (UID: \"01817576-2fb0-4596-a7ad-a940d5c458b3\") " pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:22.560961 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.560927    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:22.635705 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.635666    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:22.644676 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.644645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:22.773565 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.773458    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57b57d8c6-b8wgk"]
Jul 09 18:10:22.776312 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:10:22.776287    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70793ac_4242_4a65_83ec_92f83314d266.slice/crio-1c4bdbfaba739f631c13c74af7fe52e15d20b1c4f2400fa830fd4fadd0a82840 WatchSource:0}: Error finding container 1c4bdbfaba739f631c13c74af7fe52e15d20b1c4f2400fa830fd4fadd0a82840: Status 404 returned error can't find the container with id 1c4bdbfaba739f631c13c74af7fe52e15d20b1c4f2400fa830fd4fadd0a82840
Jul 09 18:10:22.777736 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.777716    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:10:22.794405 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.794378    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-xb5ts"]
Jul 09 18:10:22.797772 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:10:22.797749    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod01817576_2fb0_4596_a7ad_a940d5c458b3.slice/crio-1f60b6a0f26de11a4d0159cb98b6683451453193f4e9164026de43a56eaa406f WatchSource:0}: Error finding container 1f60b6a0f26de11a4d0159cb98b6683451453193f4e9164026de43a56eaa406f: Status 404 returned error can't find the container with id 1f60b6a0f26de11a4d0159cb98b6683451453193f4e9164026de43a56eaa406f
Jul 09 18:10:22.905222 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.905189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-xb5ts" event={"ID":"01817576-2fb0-4596-a7ad-a940d5c458b3","Type":"ContainerStarted","Data":"1f60b6a0f26de11a4d0159cb98b6683451453193f4e9164026de43a56eaa406f"}
Jul 09 18:10:22.906207 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:22.906184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk" event={"ID":"a70793ac-4242-4a65-83ec-92f83314d266","Type":"ContainerStarted","Data":"1c4bdbfaba739f631c13c74af7fe52e15d20b1c4f2400fa830fd4fadd0a82840"}
Jul 09 18:10:23.196986 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:23.196853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:23.294426 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:23.294387    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:26.928843 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:26.928808    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk" event={"ID":"a70793ac-4242-4a65-83ec-92f83314d266","Type":"ContainerStarted","Data":"d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2"}
Jul 09 18:10:26.929298 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:26.928969    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:26.930186 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:26.930164    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-xb5ts" event={"ID":"01817576-2fb0-4596-a7ad-a940d5c458b3","Type":"ContainerStarted","Data":"5ae05f2226498cd6de7a45002c6d380e5761f24164c0293a9ee6c06cc4d72d4f"}
Jul 09 18:10:26.930285 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:26.930276    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:26.948030 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:26.947970    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk" podStartSLOduration=1.7636094519999999 podStartE2EDuration="4.947939869s" totalImagesPullingTime="3.184330417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:26.947889885 +0000 UTC m=+371.099527149" watchObservedRunningTime="2026-07-09 18:10:26.947939869 +0000 UTC m=+371.099577132"
Jul 09 18:10:26.967511 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:26.967451    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-xb5ts" podStartSLOduration=1.767176701 podStartE2EDuration="4.967439068s" totalImagesPullingTime="3.200262367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:26.966384156 +0000 UTC m=+371.118021425" watchObservedRunningTime="2026-07-09 18:10:26.967439068 +0000 UTC m=+371.119076340"
Jul 09 18:10:41.588551 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:41.588500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:41.685475 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:41.685415    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:55.657261 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:55.657221    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:57.937226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:57.937193    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-xb5ts"
Jul 09 18:10:57.940366 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:57.940332    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:59.332474 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.332443    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-57b57d8c6-b8wgk"]
Jul 09 18:10:59.332933 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.332624    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk" podUID="a70793ac-4242-4a65-83ec-92f83314d266" containerName="manager" containerID="cri-o://d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2" gracePeriod=10
Jul 09 18:10:59.354995 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.354969    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-57b57d8c6-22rcv"]
Jul 09 18:10:59.415346 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.415323    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57b57d8c6-22rcv"]
Jul 09 18:10:59.415464 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.415429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:10:59.492813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.492787    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6sf2\" (UniqueName: \"kubernetes.io/projected/964d20ad-1d5b-43e8-9407-3248762445c5-kube-api-access-l6sf2\") pod \"kserve-controller-manager-57b57d8c6-22rcv\" (UID: \"964d20ad-1d5b-43e8-9407-3248762445c5\") " pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:10:59.492936 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.492905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/964d20ad-1d5b-43e8-9407-3248762445c5-cert\") pod \"kserve-controller-manager-57b57d8c6-22rcv\" (UID: \"964d20ad-1d5b-43e8-9407-3248762445c5\") " pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:10:59.594278 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.594247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6sf2\" (UniqueName: \"kubernetes.io/projected/964d20ad-1d5b-43e8-9407-3248762445c5-kube-api-access-l6sf2\") pod \"kserve-controller-manager-57b57d8c6-22rcv\" (UID: \"964d20ad-1d5b-43e8-9407-3248762445c5\") " pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:10:59.594396 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.594329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/964d20ad-1d5b-43e8-9407-3248762445c5-cert\") pod \"kserve-controller-manager-57b57d8c6-22rcv\" (UID: \"964d20ad-1d5b-43e8-9407-3248762445c5\") " pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:10:59.596433 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.596411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/964d20ad-1d5b-43e8-9407-3248762445c5-cert\") pod \"kserve-controller-manager-57b57d8c6-22rcv\" (UID: \"964d20ad-1d5b-43e8-9407-3248762445c5\") " pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:10:59.602024 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.601998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6sf2\" (UniqueName: \"kubernetes.io/projected/964d20ad-1d5b-43e8-9407-3248762445c5-kube-api-access-l6sf2\") pod \"kserve-controller-manager-57b57d8c6-22rcv\" (UID: \"964d20ad-1d5b-43e8-9407-3248762445c5\") " pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:10:59.631073 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.631058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:10:59.694991 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.694971    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d5xf\" (UniqueName: \"kubernetes.io/projected/a70793ac-4242-4a65-83ec-92f83314d266-kube-api-access-9d5xf\") pod \"a70793ac-4242-4a65-83ec-92f83314d266\" (UID: \"a70793ac-4242-4a65-83ec-92f83314d266\") "
Jul 09 18:10:59.695093 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.695023    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a70793ac-4242-4a65-83ec-92f83314d266-cert\") pod \"a70793ac-4242-4a65-83ec-92f83314d266\" (UID: \"a70793ac-4242-4a65-83ec-92f83314d266\") "
Jul 09 18:10:59.696918 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.696892    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70793ac-4242-4a65-83ec-92f83314d266-kube-api-access-9d5xf" (OuterVolumeSpecName: "kube-api-access-9d5xf") pod "a70793ac-4242-4a65-83ec-92f83314d266" (UID: "a70793ac-4242-4a65-83ec-92f83314d266"). InnerVolumeSpecName "kube-api-access-9d5xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:10:59.697007 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.696947    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70793ac-4242-4a65-83ec-92f83314d266-cert" (OuterVolumeSpecName: "cert") pod "a70793ac-4242-4a65-83ec-92f83314d266" (UID: "a70793ac-4242-4a65-83ec-92f83314d266"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:59.770395 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.770375    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:10:59.796427 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.796402    2560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a70793ac-4242-4a65-83ec-92f83314d266-cert\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:10:59.796427 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.796421    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d5xf\" (UniqueName: \"kubernetes.io/projected/a70793ac-4242-4a65-83ec-92f83314d266-kube-api-access-9d5xf\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:10:59.914411 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:10:59.914389    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57b57d8c6-22rcv"]
Jul 09 18:10:59.916665 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:10:59.916646    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964d20ad_1d5b_43e8_9407_3248762445c5.slice/crio-85d3955aa3d85f1cde4625632545e90969c4d07aee24a7aa36f16b0888999c5b WatchSource:0}: Error finding container 85d3955aa3d85f1cde4625632545e90969c4d07aee24a7aa36f16b0888999c5b: Status 404 returned error can't find the container with id 85d3955aa3d85f1cde4625632545e90969c4d07aee24a7aa36f16b0888999c5b
Jul 09 18:11:00.073874 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.073846    2560 generic.go:356] "Generic (PLEG): container finished" podID="a70793ac-4242-4a65-83ec-92f83314d266" containerID="d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2" exitCode=0
Jul 09 18:11:00.073982 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.073887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk" event={"ID":"a70793ac-4242-4a65-83ec-92f83314d266","Type":"ContainerDied","Data":"d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2"}
Jul 09 18:11:00.073982 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.073916    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk"
Jul 09 18:11:00.073982 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.073944    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57b57d8c6-b8wgk" event={"ID":"a70793ac-4242-4a65-83ec-92f83314d266","Type":"ContainerDied","Data":"1c4bdbfaba739f631c13c74af7fe52e15d20b1c4f2400fa830fd4fadd0a82840"}
Jul 09 18:11:00.073982 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.073960    2560 scope.go:117] "RemoveContainer" containerID="d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2"
Jul 09 18:11:00.075365 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.075331    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57b57d8c6-22rcv" event={"ID":"964d20ad-1d5b-43e8-9407-3248762445c5","Type":"ContainerStarted","Data":"85d3955aa3d85f1cde4625632545e90969c4d07aee24a7aa36f16b0888999c5b"}
Jul 09 18:11:00.082757 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.082740    2560 scope.go:117] "RemoveContainer" containerID="d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2"
Jul 09 18:11:00.082993 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:11:00.082977    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2\": container with ID starting with d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2 not found: ID does not exist" containerID="d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2"
Jul 09 18:11:00.083038 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.083000    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2"} err="failed to get container status \"d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2\": rpc error: code = NotFound desc = could not find container \"d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2\": container with ID starting with d147cf16a9c6dd1a2f0ccedbb8d35b67e505c9340a569ac7d51d4322643232b2 not found: ID does not exist"
Jul 09 18:11:00.095101 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.095014    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-57b57d8c6-b8wgk"]
Jul 09 18:11:00.096620 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.096601    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-57b57d8c6-b8wgk"]
Jul 09 18:11:00.388870 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:00.388820    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70793ac-4242-4a65-83ec-92f83314d266" path="/var/lib/kubelet/pods/a70793ac-4242-4a65-83ec-92f83314d266/volumes"
Jul 09 18:11:01.082878 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:01.082841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57b57d8c6-22rcv" event={"ID":"964d20ad-1d5b-43e8-9407-3248762445c5","Type":"ContainerStarted","Data":"3483039c3ae2c854db4877237e85cfa20b188507e1b18aabe915f1b4a7a24754"}
Jul 09 18:11:01.083110 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:01.082991    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:11:01.101575 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:01.101526    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-57b57d8c6-22rcv" podStartSLOduration=1.5194976759999999 podStartE2EDuration="2.101509542s" totalImagesPullingTime="582.011866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:01.09978127 +0000 UTC m=+405.251418543" watchObservedRunningTime="2026-07-09 18:11:01.101509542 +0000 UTC m=+405.253146814"
Jul 09 18:11:06.696672 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:06.696630    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845789c8d5-kxvv6"]
Jul 09 18:11:06.891642 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:06.891607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:06.994593 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:06.994564    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:07.065821 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:07.065789    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:31.716506 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:31.716450    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-845789c8d5-kxvv6" podUID="c4490d50-1057-4942-9ee2-847b334c3241" containerName="console" containerID="cri-o://d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b" gracePeriod=15
Jul 09 18:11:31.949145 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:31.949123    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:11:32.028059 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028035    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-service-ca\") pod \"c4490d50-1057-4942-9ee2-847b334c3241\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") "
Jul 09 18:11:32.028170 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028062    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-trusted-ca-bundle\") pod \"c4490d50-1057-4942-9ee2-847b334c3241\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") "
Jul 09 18:11:32.028170 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028113    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-oauth-serving-cert\") pod \"c4490d50-1057-4942-9ee2-847b334c3241\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") "
Jul 09 18:11:32.028282 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028231    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-oauth-config\") pod \"c4490d50-1057-4942-9ee2-847b334c3241\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") "
Jul 09 18:11:32.028282 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028272    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-console-config\") pod \"c4490d50-1057-4942-9ee2-847b334c3241\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") "
Jul 09 18:11:32.028441 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028305    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-serving-cert\") pod \"c4490d50-1057-4942-9ee2-847b334c3241\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") "
Jul 09 18:11:32.028441 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028335    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76j48\" (UniqueName: \"kubernetes.io/projected/c4490d50-1057-4942-9ee2-847b334c3241-kube-api-access-76j48\") pod \"c4490d50-1057-4942-9ee2-847b334c3241\" (UID: \"c4490d50-1057-4942-9ee2-847b334c3241\") "
Jul 09 18:11:32.028575 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028448    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c4490d50-1057-4942-9ee2-847b334c3241" (UID: "c4490d50-1057-4942-9ee2-847b334c3241"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:32.028575 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028463    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-service-ca" (OuterVolumeSpecName: "service-ca") pod "c4490d50-1057-4942-9ee2-847b334c3241" (UID: "c4490d50-1057-4942-9ee2-847b334c3241"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:32.028682 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028588    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c4490d50-1057-4942-9ee2-847b334c3241" (UID: "c4490d50-1057-4942-9ee2-847b334c3241"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:32.028682 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028647    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-console-config" (OuterVolumeSpecName: "console-config") pod "c4490d50-1057-4942-9ee2-847b334c3241" (UID: "c4490d50-1057-4942-9ee2-847b334c3241"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:32.028779 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028755    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-service-ca\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:11:32.028779 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028768    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-trusted-ca-bundle\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:11:32.028779 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028776    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-oauth-serving-cert\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:11:32.028869 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.028785    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4490d50-1057-4942-9ee2-847b334c3241-console-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:11:32.030325 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.030302    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c4490d50-1057-4942-9ee2-847b334c3241" (UID: "c4490d50-1057-4942-9ee2-847b334c3241"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:32.030415 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.030349    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c4490d50-1057-4942-9ee2-847b334c3241" (UID: "c4490d50-1057-4942-9ee2-847b334c3241"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:32.030415 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.030357    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4490d50-1057-4942-9ee2-847b334c3241-kube-api-access-76j48" (OuterVolumeSpecName: "kube-api-access-76j48") pod "c4490d50-1057-4942-9ee2-847b334c3241" (UID: "c4490d50-1057-4942-9ee2-847b334c3241"). InnerVolumeSpecName "kube-api-access-76j48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:11:32.093808 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.093784    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-57b57d8c6-22rcv"
Jul 09 18:11:32.130135 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.130109    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-oauth-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:11:32.130135 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.130136    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4490d50-1057-4942-9ee2-847b334c3241-console-serving-cert\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:11:32.130281 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.130149    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76j48\" (UniqueName: \"kubernetes.io/projected/c4490d50-1057-4942-9ee2-847b334c3241-kube-api-access-76j48\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:11:32.226542 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.226515    2560 generic.go:356] "Generic (PLEG): container finished" podID="c4490d50-1057-4942-9ee2-847b334c3241" containerID="d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b" exitCode=0
Jul 09 18:11:32.226636 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.226544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845789c8d5-kxvv6" event={"ID":"c4490d50-1057-4942-9ee2-847b334c3241","Type":"ContainerDied","Data":"d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b"}
Jul 09 18:11:32.226636 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.226567    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845789c8d5-kxvv6"
Jul 09 18:11:32.226636 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.226595    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845789c8d5-kxvv6" event={"ID":"c4490d50-1057-4942-9ee2-847b334c3241","Type":"ContainerDied","Data":"da6c3a4d45fafa18896d90e90ac183a8a98ae0bc2375d170d78ce0dcb2d825f4"}
Jul 09 18:11:32.226636 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.226616    2560 scope.go:117] "RemoveContainer" containerID="d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b"
Jul 09 18:11:32.235092 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.235077    2560 scope.go:117] "RemoveContainer" containerID="d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b"
Jul 09 18:11:32.235322 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:11:32.235305    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b\": container with ID starting with d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b not found: ID does not exist" containerID="d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b"
Jul 09 18:11:32.235371 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.235329    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b"} err="failed to get container status \"d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b\": rpc error: code = NotFound desc = could not find container \"d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b\": container with ID starting with d45158d8f8bb0d7958f7f4d4f335c77e57e711edeeb668f1ac1a2d02c0a9760b not found: ID does not exist"
Jul 09 18:11:32.258072 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.258050    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845789c8d5-kxvv6"]
Jul 09 18:11:32.264352 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.264333    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-845789c8d5-kxvv6"]
Jul 09 18:11:32.388033 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:32.387982    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4490d50-1057-4942-9ee2-847b334c3241" path="/var/lib/kubelet/pods/c4490d50-1057-4942-9ee2-847b334c3241/volumes"
Jul 09 18:11:33.121623 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.121591    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:33.180007 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.179982    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-sw9vg"]
Jul 09 18:11:33.180356 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.180339    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70793ac-4242-4a65-83ec-92f83314d266" containerName="manager"
Jul 09 18:11:33.180417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.180356    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70793ac-4242-4a65-83ec-92f83314d266" containerName="manager"
Jul 09 18:11:33.180417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.180388    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4490d50-1057-4942-9ee2-847b334c3241" containerName="console"
Jul 09 18:11:33.180417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.180397    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4490d50-1057-4942-9ee2-847b334c3241" containerName="console"
Jul 09 18:11:33.180620 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.180581    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70793ac-4242-4a65-83ec-92f83314d266" containerName="manager"
Jul 09 18:11:33.180620 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.180594    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4490d50-1057-4942-9ee2-847b334c3241" containerName="console"
Jul 09 18:11:33.185514 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.185499    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:33.188500 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.188469    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-cmb6j\""
Jul 09 18:11:33.188634 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.188618    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 09 18:11:33.195193 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.195174    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-trcc8"]
Jul 09 18:11:33.198735 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.198717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:33.198933 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.198913    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-sw9vg"]
Jul 09 18:11:33.202652 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.202634    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-snh6q\""
Jul 09 18:11:33.203091 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.203074    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 09 18:11:33.214255 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.214238    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-trcc8"]
Jul 09 18:11:33.338722 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.338698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a92932da-b8c0-4eb9-926b-b960ddf5feb7-cert\") pod \"odh-model-controller-84b54b7f97-trcc8\" (UID: \"a92932da-b8c0-4eb9-926b-b960ddf5feb7\") " pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:33.338813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.338728    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3533f2-0ecf-442f-9ea4-9c1fc338a031-tls-certs\") pod \"model-serving-api-fd65d7d6-sw9vg\" (UID: \"9f3533f2-0ecf-442f-9ea4-9c1fc338a031\") " pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:33.338813 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.338774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhvw\" (UniqueName: \"kubernetes.io/projected/9f3533f2-0ecf-442f-9ea4-9c1fc338a031-kube-api-access-mvhvw\") pod \"model-serving-api-fd65d7d6-sw9vg\" (UID: \"9f3533f2-0ecf-442f-9ea4-9c1fc338a031\") " pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:33.338880 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.338845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdd6l\" (UniqueName: \"kubernetes.io/projected/a92932da-b8c0-4eb9-926b-b960ddf5feb7-kube-api-access-xdd6l\") pod \"odh-model-controller-84b54b7f97-trcc8\" (UID: \"a92932da-b8c0-4eb9-926b-b960ddf5feb7\") " pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:33.439978 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.439915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhvw\" (UniqueName: \"kubernetes.io/projected/9f3533f2-0ecf-442f-9ea4-9c1fc338a031-kube-api-access-mvhvw\") pod \"model-serving-api-fd65d7d6-sw9vg\" (UID: \"9f3533f2-0ecf-442f-9ea4-9c1fc338a031\") " pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:33.439978 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.439957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdd6l\" (UniqueName: \"kubernetes.io/projected/a92932da-b8c0-4eb9-926b-b960ddf5feb7-kube-api-access-xdd6l\") pod \"odh-model-controller-84b54b7f97-trcc8\" (UID: \"a92932da-b8c0-4eb9-926b-b960ddf5feb7\") " pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:33.440106 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.440088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a92932da-b8c0-4eb9-926b-b960ddf5feb7-cert\") pod \"odh-model-controller-84b54b7f97-trcc8\" (UID: \"a92932da-b8c0-4eb9-926b-b960ddf5feb7\") " pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:33.440142 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.440126    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3533f2-0ecf-442f-9ea4-9c1fc338a031-tls-certs\") pod \"model-serving-api-fd65d7d6-sw9vg\" (UID: \"9f3533f2-0ecf-442f-9ea4-9c1fc338a031\") " pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:33.440187 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:11:33.440167    2560 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 18:11:33.440247 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:11:33.440235    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92932da-b8c0-4eb9-926b-b960ddf5feb7-cert podName:a92932da-b8c0-4eb9-926b-b960ddf5feb7 nodeName:}" failed. No retries permitted until 2026-07-09 18:11:33.940219258 +0000 UTC m=+438.091856507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a92932da-b8c0-4eb9-926b-b960ddf5feb7-cert") pod "odh-model-controller-84b54b7f97-trcc8" (UID: "a92932da-b8c0-4eb9-926b-b960ddf5feb7") : secret "odh-model-controller-webhook-cert" not found
Jul 09 18:11:33.442747 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.442727    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3533f2-0ecf-442f-9ea4-9c1fc338a031-tls-certs\") pod \"model-serving-api-fd65d7d6-sw9vg\" (UID: \"9f3533f2-0ecf-442f-9ea4-9c1fc338a031\") " pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:33.454240 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.454207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhvw\" (UniqueName: \"kubernetes.io/projected/9f3533f2-0ecf-442f-9ea4-9c1fc338a031-kube-api-access-mvhvw\") pod \"model-serving-api-fd65d7d6-sw9vg\" (UID: \"9f3533f2-0ecf-442f-9ea4-9c1fc338a031\") " pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:33.454408 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.454389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdd6l\" (UniqueName: \"kubernetes.io/projected/a92932da-b8c0-4eb9-926b-b960ddf5feb7-kube-api-access-xdd6l\") pod \"odh-model-controller-84b54b7f97-trcc8\" (UID: \"a92932da-b8c0-4eb9-926b-b960ddf5feb7\") " pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:33.496036 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.496014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:33.632221 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.632192    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-sw9vg"]
Jul 09 18:11:33.634679 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:11:33.634653    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3533f2_0ecf_442f_9ea4_9c1fc338a031.slice/crio-c8382fc86afa667a48e2356257b2a0aac971a918569d15506bf3513ba19c27b3 WatchSource:0}: Error finding container c8382fc86afa667a48e2356257b2a0aac971a918569d15506bf3513ba19c27b3: Status 404 returned error can't find the container with id c8382fc86afa667a48e2356257b2a0aac971a918569d15506bf3513ba19c27b3
Jul 09 18:11:33.945323 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.945284    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a92932da-b8c0-4eb9-926b-b960ddf5feb7-cert\") pod \"odh-model-controller-84b54b7f97-trcc8\" (UID: \"a92932da-b8c0-4eb9-926b-b960ddf5feb7\") " pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:33.947588 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:33.947569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a92932da-b8c0-4eb9-926b-b960ddf5feb7-cert\") pod \"odh-model-controller-84b54b7f97-trcc8\" (UID: \"a92932da-b8c0-4eb9-926b-b960ddf5feb7\") " pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:34.108512 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:34.108468    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:34.159584 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:34.159531    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:34.236664 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:34.236591    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-sw9vg" event={"ID":"9f3533f2-0ecf-442f-9ea4-9c1fc338a031","Type":"ContainerStarted","Data":"c8382fc86afa667a48e2356257b2a0aac971a918569d15506bf3513ba19c27b3"}
Jul 09 18:11:34.249586 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:34.249563    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-trcc8"]
Jul 09 18:11:34.258152 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:34.258124    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:35.242921 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:35.242884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-trcc8" event={"ID":"a92932da-b8c0-4eb9-926b-b960ddf5feb7","Type":"ContainerStarted","Data":"39725d4718e4ae4ce5efe4b340cda70dcda928fdcb376e89ba8cecc37c43167f"}
Jul 09 18:11:38.259868 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:38.259793    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-trcc8" event={"ID":"a92932da-b8c0-4eb9-926b-b960ddf5feb7","Type":"ContainerStarted","Data":"e8c4844a3aedc6293e0ad77aa3a464688764018ea8729c676d355bc49f7a19da"}
Jul 09 18:11:38.259868 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:38.259860    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:38.261170 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:38.261149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-sw9vg" event={"ID":"9f3533f2-0ecf-442f-9ea4-9c1fc338a031","Type":"ContainerStarted","Data":"076175a8e9962ea539801638754ea33cf5477656930259e37681fe1890b037bb"}
Jul 09 18:11:38.261273 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:38.261198    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:38.285436 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:38.285394    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-trcc8" podStartSLOduration=2.17762214 podStartE2EDuration="5.285382191s" totalImagesPullingTime="3.107760051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:38.283765582 +0000 UTC m=+442.435402874" watchObservedRunningTime="2026-07-09 18:11:38.285382191 +0000 UTC m=+442.437019441"
Jul 09 18:11:38.307117 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:38.307084    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-sw9vg" podStartSLOduration=1.5824910760000002 podStartE2EDuration="5.307072652s" totalImagesPullingTime="3.724581576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:38.30640761 +0000 UTC m=+442.458044893" watchObservedRunningTime="2026-07-09 18:11:38.307072652 +0000 UTC m=+442.458709923"
Jul 09 18:11:49.267982 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:49.267942    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-trcc8"
Jul 09 18:11:49.269922 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:49.269894    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-sw9vg"
Jul 09 18:11:59.991121 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:11:59.991082    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:02.014650 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:12:02.014600    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:12:59.976930 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:12:59.976890    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:09.470328 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:09.470225    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:09.565960 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:09.565928    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:21.141283 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.141243    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:21.248410 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.248378    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:21.341714 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.341681    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:21.472227 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.472150    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:21.633123 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.633090    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:21.716381 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.716346    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"]
Jul 09 18:13:21.719791 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.719770    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:21.722542 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.722471    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fgcf4\""
Jul 09 18:13:21.722542 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.722514    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7eb51-predictor-serving-cert\""
Jul 09 18:13:21.722542 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.722521    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:13:21.723262 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.723248    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:13:21.723341 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.723252    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7eb51-kube-rbac-proxy-sar-config\""
Jul 09 18:13:21.727658 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.727631    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:21.737196 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.737177    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"]
Jul 09 18:13:21.836917 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.836888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g45c\" (UniqueName: \"kubernetes.io/projected/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-api-access-8g45c\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:21.837040 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.836921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:21.837040 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.836942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13756b7a-3b4b-4be7-8332-52e70770ece2-proxy-tls\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:21.864252 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.864205    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:21.938197 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.938170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g45c\" (UniqueName: \"kubernetes.io/projected/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-api-access-8g45c\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:21.938330 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.938203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:21.938330 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.938226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13756b7a-3b4b-4be7-8332-52e70770ece2-proxy-tls\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:21.938330 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:13:21.938319    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-serving-cert: secret "error-404-isvc-7eb51-predictor-serving-cert" not found
Jul 09 18:13:21.938519 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:13:21.938376    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13756b7a-3b4b-4be7-8332-52e70770ece2-proxy-tls podName:13756b7a-3b4b-4be7-8332-52e70770ece2 nodeName:}" failed. No retries permitted until 2026-07-09 18:13:22.438357099 +0000 UTC m=+546.589994362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/13756b7a-3b4b-4be7-8332-52e70770ece2-proxy-tls") pod "error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" (UID: "13756b7a-3b4b-4be7-8332-52e70770ece2") : secret "error-404-isvc-7eb51-predictor-serving-cert" not found
Jul 09 18:13:21.938919 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.938895    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:21.974836 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:21.974786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g45c\" (UniqueName: \"kubernetes.io/projected/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-api-access-8g45c\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:22.090414 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:22.090384    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:22.198036 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:22.198008    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:22.441010 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:22.440976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13756b7a-3b4b-4be7-8332-52e70770ece2-proxy-tls\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:22.443455 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:22.443433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13756b7a-3b4b-4be7-8332-52e70770ece2-proxy-tls\") pod \"error-404-isvc-7eb51-predictor-695f658b7c-cmcvh\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") " pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:22.631009 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:22.630988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:22.768077 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:22.768052    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"]
Jul 09 18:13:22.769424 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:13:22.769402    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13756b7a_3b4b_4be7_8332_52e70770ece2.slice/crio-2e41ea4861106310c7d3d575be7d4505ac79090b9d05cb2cfe61a29d1d8ca170 WatchSource:0}: Error finding container 2e41ea4861106310c7d3d575be7d4505ac79090b9d05cb2cfe61a29d1d8ca170: Status 404 returned error can't find the container with id 2e41ea4861106310c7d3d575be7d4505ac79090b9d05cb2cfe61a29d1d8ca170
Jul 09 18:13:23.704694 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:23.704656    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" event={"ID":"13756b7a-3b4b-4be7-8332-52e70770ece2","Type":"ContainerStarted","Data":"2e41ea4861106310c7d3d575be7d4505ac79090b9d05cb2cfe61a29d1d8ca170"}
Jul 09 18:13:36.768611 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:36.768572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" event={"ID":"13756b7a-3b4b-4be7-8332-52e70770ece2","Type":"ContainerStarted","Data":"f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e"}
Jul 09 18:13:38.779206 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:38.779172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" event={"ID":"13756b7a-3b4b-4be7-8332-52e70770ece2","Type":"ContainerStarted","Data":"4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea"}
Jul 09 18:13:38.779604 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:38.779374    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:38.779604 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:38.779507    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:38.780648 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:38.780618    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:13:38.807677 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:38.807628    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" podStartSLOduration=2.42759697 podStartE2EDuration="17.807610487s" totalImagesPullingTime="15.380013517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:38.803732899 +0000 UTC m=+562.955370172" watchObservedRunningTime="2026-07-09 18:13:38.807610487 +0000 UTC m=+562.959247760"
Jul 09 18:13:39.783773 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:39.783737    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:13:44.788254 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:44.788225    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:13:44.788837 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:44.788721    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:13:54.788904 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:13:54.788864    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:14:04.788762 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:14:04.788728    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:14:14.789107 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:14:14.789058    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 09 18:14:16.252093 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:14:16.252067    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:14:16.252726 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:14:16.252706    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:14:24.790294 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:14:24.790259    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:14:31.609473 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:14:31.609429    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:01.147927 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:01.147837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:01.366226 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:01.366189    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:11.369538 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.369503    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:11.603825 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.603785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:11.681785 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.681711    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:11.721779 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.721755    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"]
Jul 09 18:15:11.724037 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.724020    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:11.727397 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.727369    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ea8a9-predictor-serving-cert\""
Jul 09 18:15:11.727530 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.727401    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ea8a9-kube-rbac-proxy-sar-config\""
Jul 09 18:15:11.735891 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.735867    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"]
Jul 09 18:15:11.756631 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.756607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:11.770452 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.770431    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"]
Jul 09 18:15:11.770661 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.770637    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container" containerID="cri-o://f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e" gracePeriod=30
Jul 09 18:15:11.770752 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.770717    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kube-rbac-proxy" containerID="cri-o://4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea" gracePeriod=30
Jul 09 18:15:11.816922 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.816897    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:11.890243 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.890223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:11.890339 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.890259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5cv\" (UniqueName: \"kubernetes.io/projected/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-api-access-rz5cv\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:11.890339 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.890327    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0489a9a6-a24c-415c-b33a-9174a389c8f1-proxy-tls\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:11.990893 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.990865    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:11.991008 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.990908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5cv\" (UniqueName: \"kubernetes.io/projected/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-api-access-rz5cv\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:11.991008 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.990970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0489a9a6-a24c-415c-b33a-9174a389c8f1-proxy-tls\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:11.991136 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:15:11.991095    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-serving-cert: secret "error-404-isvc-ea8a9-predictor-serving-cert" not found
Jul 09 18:15:11.991193 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:15:11.991165    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0489a9a6-a24c-415c-b33a-9174a389c8f1-proxy-tls podName:0489a9a6-a24c-415c-b33a-9174a389c8f1 nodeName:}" failed. No retries permitted until 2026-07-09 18:15:12.49114404 +0000 UTC m=+656.642781290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0489a9a6-a24c-415c-b33a-9174a389c8f1-proxy-tls") pod "error-404-isvc-ea8a9-predictor-74877969-qprj9" (UID: "0489a9a6-a24c-415c-b33a-9174a389c8f1") : secret "error-404-isvc-ea8a9-predictor-serving-cert" not found
Jul 09 18:15:11.991572 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:11.991554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:12.003097 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:12.003067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5cv\" (UniqueName: \"kubernetes.io/projected/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-api-access-rz5cv\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:12.040271 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:12.040249    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:12.175018 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:12.174990    2560 generic.go:356] "Generic (PLEG): container finished" podID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerID="4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea" exitCode=2
Jul 09 18:15:12.175115 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:12.175068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" event={"ID":"13756b7a-3b4b-4be7-8332-52e70770ece2","Type":"ContainerDied","Data":"4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea"}
Jul 09 18:15:12.386949 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:12.386867    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:12.494716 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:12.494692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0489a9a6-a24c-415c-b33a-9174a389c8f1-proxy-tls\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:12.496949 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:12.496926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0489a9a6-a24c-415c-b33a-9174a389c8f1-proxy-tls\") pod \"error-404-isvc-ea8a9-predictor-74877969-qprj9\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") " pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:12.634437 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:12.634409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:12.764166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:12.764095    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"]
Jul 09 18:15:12.766309 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:15:12.766284    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0489a9a6_a24c_415c_b33a_9174a389c8f1.slice/crio-092d2c4cb73ff3ac0aa88713ee886b7a743e3560ae156896d20f04de201e2ed7 WatchSource:0}: Error finding container 092d2c4cb73ff3ac0aa88713ee886b7a743e3560ae156896d20f04de201e2ed7: Status 404 returned error can't find the container with id 092d2c4cb73ff3ac0aa88713ee886b7a743e3560ae156896d20f04de201e2ed7
Jul 09 18:15:13.181914 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:13.181878    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" event={"ID":"0489a9a6-a24c-415c-b33a-9174a389c8f1","Type":"ContainerStarted","Data":"2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53"}
Jul 09 18:15:13.182121 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:13.181941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" event={"ID":"0489a9a6-a24c-415c-b33a-9174a389c8f1","Type":"ContainerStarted","Data":"3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea"}
Jul 09 18:15:13.182121 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:13.181989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" event={"ID":"0489a9a6-a24c-415c-b33a-9174a389c8f1","Type":"ContainerStarted","Data":"092d2c4cb73ff3ac0aa88713ee886b7a743e3560ae156896d20f04de201e2ed7"}
Jul 09 18:15:13.182121 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:13.182024    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:13.202141 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:13.202089    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" podStartSLOduration=2.202073191 podStartE2EDuration="2.202073191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:13.20095428 +0000 UTC m=+657.352591553" watchObservedRunningTime="2026-07-09 18:15:13.202073191 +0000 UTC m=+657.353710462"
Jul 09 18:15:14.187237 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.187199    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:14.189126 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.188891    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:15:14.413639 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.413618    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:15:14.509881 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.509855    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-rbac-proxy-sar-config\") pod \"13756b7a-3b4b-4be7-8332-52e70770ece2\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") "
Jul 09 18:15:14.510013 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.509896    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13756b7a-3b4b-4be7-8332-52e70770ece2-proxy-tls\") pod \"13756b7a-3b4b-4be7-8332-52e70770ece2\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") "
Jul 09 18:15:14.510013 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.509934    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g45c\" (UniqueName: \"kubernetes.io/projected/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-api-access-8g45c\") pod \"13756b7a-3b4b-4be7-8332-52e70770ece2\" (UID: \"13756b7a-3b4b-4be7-8332-52e70770ece2\") "
Jul 09 18:15:14.510244 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.510212    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13756b7a-3b4b-4be7-8332-52e70770ece2" (UID: "13756b7a-3b4b-4be7-8332-52e70770ece2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:15:14.510393 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.510375    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:15:14.511994 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.511967    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13756b7a-3b4b-4be7-8332-52e70770ece2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13756b7a-3b4b-4be7-8332-52e70770ece2" (UID: "13756b7a-3b4b-4be7-8332-52e70770ece2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:15:14.512083 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.512068    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-api-access-8g45c" (OuterVolumeSpecName: "kube-api-access-8g45c") pod "13756b7a-3b4b-4be7-8332-52e70770ece2" (UID: "13756b7a-3b4b-4be7-8332-52e70770ece2"). InnerVolumeSpecName "kube-api-access-8g45c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:15:14.611597 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.611572    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13756b7a-3b4b-4be7-8332-52e70770ece2-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:15:14.611597 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:14.611595    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g45c\" (UniqueName: \"kubernetes.io/projected/13756b7a-3b4b-4be7-8332-52e70770ece2-kube-api-access-8g45c\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:15:15.193025 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.192995    2560 generic.go:356] "Generic (PLEG): container finished" podID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerID="f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e" exitCode=0
Jul 09 18:15:15.193375 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.193051    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"
Jul 09 18:15:15.193375 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.193080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" event={"ID":"13756b7a-3b4b-4be7-8332-52e70770ece2","Type":"ContainerDied","Data":"f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e"}
Jul 09 18:15:15.193375 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.193143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh" event={"ID":"13756b7a-3b4b-4be7-8332-52e70770ece2","Type":"ContainerDied","Data":"2e41ea4861106310c7d3d575be7d4505ac79090b9d05cb2cfe61a29d1d8ca170"}
Jul 09 18:15:15.193596 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.193569    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:15:15.194352 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.193166    2560 scope.go:117] "RemoveContainer" containerID="4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea"
Jul 09 18:15:15.202800 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.202785    2560 scope.go:117] "RemoveContainer" containerID="f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e"
Jul 09 18:15:15.211330 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.211315    2560 scope.go:117] "RemoveContainer" containerID="4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea"
Jul 09 18:15:15.211597 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:15:15.211577    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea\": container with ID starting with 4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea not found: ID does not exist" containerID="4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea"
Jul 09 18:15:15.211676 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.211608    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea"} err="failed to get container status \"4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea\": rpc error: code = NotFound desc = could not find container \"4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea\": container with ID starting with 4b0feff9131f160efb5ff624607f827008ef26a3115ddb6f1c0900f899eb2bea not found: ID does not exist"
Jul 09 18:15:15.211676 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.211630    2560 scope.go:117] "RemoveContainer" containerID="f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e"
Jul 09 18:15:15.211876 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:15:15.211859    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e\": container with ID starting with f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e not found: ID does not exist" containerID="f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e"
Jul 09 18:15:15.211927 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.211883    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e"} err="failed to get container status \"f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e\": rpc error: code = NotFound desc = could not find container \"f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e\": container with ID starting with f372e6fc3dc241610f65142e427c645ca1109b73b04684b62f2950cd1703560e not found: ID does not exist"
Jul 09 18:15:15.214586 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.214565    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"]
Jul 09 18:15:15.218280 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:15.218262    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7eb51-predictor-695f658b7c-cmcvh"]
Jul 09 18:15:16.388641 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:16.388614    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" path="/var/lib/kubelet/pods/13756b7a-3b4b-4be7-8332-52e70770ece2/volumes"
Jul 09 18:15:20.198373 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:20.198348    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:15:20.198856 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:20.198821    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:15:30.199335 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:30.199297    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:15:31.101979 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:31.101944    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:31.298231 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:31.298193    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:40.199548 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:40.199510    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:15:41.207198 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.207167    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:41.409423 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.409377    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:41.485962 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.485881    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:41.490932 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.490889    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"]
Jul 09 18:15:41.491303 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.491288    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kube-rbac-proxy"
Jul 09 18:15:41.491409 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.491307    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kube-rbac-proxy"
Jul 09 18:15:41.491409 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.491333    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container"
Jul 09 18:15:41.491409 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.491341    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container"
Jul 09 18:15:41.491597 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.491423    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kube-rbac-proxy"
Jul 09 18:15:41.491597 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.491443    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="13756b7a-3b4b-4be7-8332-52e70770ece2" containerName="kserve-container"
Jul 09 18:15:41.493426 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.493408    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:41.496008 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.495978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-04bd4-predictor-serving-cert\""
Jul 09 18:15:41.496146 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.496012    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-04bd4-kube-rbac-proxy-sar-config\""
Jul 09 18:15:41.505599 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.505575    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"]
Jul 09 18:15:41.553223 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.553175    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:41.618245 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.618218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlf57\" (UniqueName: \"kubernetes.io/projected/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-api-access-jlf57\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:41.618414 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.618260    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-proxy-tls\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:41.618414 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.618298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:41.661613 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.661570    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:41.719093 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.719062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-proxy-tls\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:41.719256 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.719109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:41.719256 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.719166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlf57\" (UniqueName: \"kubernetes.io/projected/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-api-access-jlf57\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:41.719256 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:15:41.719223    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-serving-cert: secret "error-404-isvc-04bd4-predictor-serving-cert" not found
Jul 09 18:15:41.719421 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:15:41.719291    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-proxy-tls podName:52c2d7ec-855f-42a1-9c52-7dd4c2b720c2 nodeName:}" failed. No retries permitted until 2026-07-09 18:15:42.219272244 +0000 UTC m=+686.370909495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-proxy-tls") pod "error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" (UID: "52c2d7ec-855f-42a1-9c52-7dd4c2b720c2") : secret "error-404-isvc-04bd4-predictor-serving-cert" not found
Jul 09 18:15:41.719868 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.719845    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:41.728238 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.728214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlf57\" (UniqueName: \"kubernetes.io/projected/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-api-access-jlf57\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:41.903332 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:41.903282    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:42.222538 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:42.222423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-proxy-tls\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:42.224819 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:42.224786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-proxy-tls\") pod \"error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") " pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:42.404824 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:42.404790    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:42.535926 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:42.535901    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"]
Jul 09 18:15:42.538272 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:15:42.538248    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c2d7ec_855f_42a1_9c52_7dd4c2b720c2.slice/crio-bbe6620ae84e23170a1f69fc5e06f0f2900fed5ff2a721108df0351f75d8ce55 WatchSource:0}: Error finding container bbe6620ae84e23170a1f69fc5e06f0f2900fed5ff2a721108df0351f75d8ce55: Status 404 returned error can't find the container with id bbe6620ae84e23170a1f69fc5e06f0f2900fed5ff2a721108df0351f75d8ce55
Jul 09 18:15:43.318986 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:43.318948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" event={"ID":"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2","Type":"ContainerStarted","Data":"f9c239b1061911344ee06a85de9ecdc38449bec7b750b3cdd4987a97a4c23334"}
Jul 09 18:15:43.319428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:43.319020    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:43.319428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:43.319051    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:43.319428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:43.319076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" event={"ID":"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2","Type":"ContainerStarted","Data":"625b66987444c8341eea03d4780e3bea9e9bd05b85e19f773a428eef204edaa5"}
Jul 09 18:15:43.319428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:43.319106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" event={"ID":"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2","Type":"ContainerStarted","Data":"bbe6620ae84e23170a1f69fc5e06f0f2900fed5ff2a721108df0351f75d8ce55"}
Jul 09 18:15:43.320568 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:43.320542    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:15:43.339421 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:43.339369    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" podStartSLOduration=2.339355203 podStartE2EDuration="2.339355203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:43.338052062 +0000 UTC m=+687.489689332" watchObservedRunningTime="2026-07-09 18:15:43.339355203 +0000 UTC m=+687.490992478"
Jul 09 18:15:44.323284 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:44.323244    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:15:49.328384 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:49.328356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:15:49.328865 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:49.328838    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:15:50.199444 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:50.199406    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 09 18:15:59.329325 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:15:59.329286    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:16:00.200079 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:16:00.200054    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:16:09.329341 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:16:09.329265    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:16:11.630354 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:16:11.630310    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:13.240986 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:16:13.240950    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:19.328960 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:16:19.328906    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 09 18:16:29.329664 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:16:29.329632    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:16:41.427699 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:16:41.427663    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:41.611277 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:16:41.611236    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:07.035698 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:17:07.035657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:47.209452 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:18:47.209417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:18:47.279459 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:18:47.279428    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:16.278844 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:19:16.278773    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:19:16.279909 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:19:16.279889    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:20:42.642772 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:20:42.642660    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:20:59.767241 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:20:59.767171    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:23:36.253960 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:23:36.253911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:16.310062 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:16.309976    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:24:16.312410 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:16.312388    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:24:26.192066 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.192032    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:26.360368 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.360335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:26.420686 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.420655    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"]
Jul 09 18:24:26.420924 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.420887    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container" containerID="cri-o://3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea" gracePeriod=30
Jul 09 18:24:26.420924 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.420910    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kube-rbac-proxy" containerID="cri-o://2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53" gracePeriod=30
Jul 09 18:24:26.456067 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.456010    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"]
Jul 09 18:24:26.459417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.459396    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.461862 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.461832    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d3ac8-kube-rbac-proxy-sar-config\""
Jul 09 18:24:26.461862 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.461845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:26.462032 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.461921    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d3ac8-predictor-serving-cert\""
Jul 09 18:24:26.469387 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.469366    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"]
Jul 09 18:24:26.488183 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.488162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxjx\" (UniqueName: \"kubernetes.io/projected/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-api-access-2wxjx\") pod \"error-404-isvc-d3ac8-predictor-59486dd895-2dj4n\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") " pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.488276 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.488193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7921fc-9596-4ec7-a2d2-da1101506d46-proxy-tls\") pod \"error-404-isvc-d3ac8-predictor-59486dd895-2dj4n\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") " pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.488276 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.488234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d3ac8-predictor-59486dd895-2dj4n\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") " pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.555186 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.555161    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:26.589596 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.589570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d3ac8-predictor-59486dd895-2dj4n\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") " pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.589713 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.589668    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxjx\" (UniqueName: \"kubernetes.io/projected/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-api-access-2wxjx\") pod \"error-404-isvc-d3ac8-predictor-59486dd895-2dj4n\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") " pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.589713 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.589698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7921fc-9596-4ec7-a2d2-da1101506d46-proxy-tls\") pod \"error-404-isvc-d3ac8-predictor-59486dd895-2dj4n\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") " pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.590245 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.590217    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d3ac8-predictor-59486dd895-2dj4n\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") " pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.591942 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.591916    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7921fc-9596-4ec7-a2d2-da1101506d46-proxy-tls\") pod \"error-404-isvc-d3ac8-predictor-59486dd895-2dj4n\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") " pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.598180 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.598149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxjx\" (UniqueName: \"kubernetes.io/projected/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-api-access-2wxjx\") pod \"error-404-isvc-d3ac8-predictor-59486dd895-2dj4n\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") " pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.601391 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.601367    2560 generic.go:356] "Generic (PLEG): container finished" podID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerID="2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53" exitCode=2
Jul 09 18:24:26.601514 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.601442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" event={"ID":"0489a9a6-a24c-415c-b33a-9174a389c8f1","Type":"ContainerDied","Data":"2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53"}
Jul 09 18:24:26.649648 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.649615    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:26.770882 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.770859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:26.919890 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:26.919828    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"]
Jul 09 18:24:26.922536 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:24:26.922508    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7921fc_9596_4ec7_a2d2_da1101506d46.slice/crio-fe6304861e0d72be595c0e8093466fcf4f57686131defd84ce131198d4386288 WatchSource:0}: Error finding container fe6304861e0d72be595c0e8093466fcf4f57686131defd84ce131198d4386288: Status 404 returned error can't find the container with id fe6304861e0d72be595c0e8093466fcf4f57686131defd84ce131198d4386288
Jul 09 18:24:27.607300 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:27.607270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" event={"ID":"ef7921fc-9596-4ec7-a2d2-da1101506d46","Type":"ContainerStarted","Data":"6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f"}
Jul 09 18:24:27.607774 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:27.607328    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" event={"ID":"ef7921fc-9596-4ec7-a2d2-da1101506d46","Type":"ContainerStarted","Data":"23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a"}
Jul 09 18:24:27.607774 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:27.607354    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" event={"ID":"ef7921fc-9596-4ec7-a2d2-da1101506d46","Type":"ContainerStarted","Data":"fe6304861e0d72be595c0e8093466fcf4f57686131defd84ce131198d4386288"}
Jul 09 18:24:27.607774 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:27.607375    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:27.628807 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:27.628759    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podStartSLOduration=1.628744143 podStartE2EDuration="1.628744143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:27.626867896 +0000 UTC m=+1211.778505201" watchObservedRunningTime="2026-07-09 18:24:27.628744143 +0000 UTC m=+1211.780381415"
Jul 09 18:24:28.611093 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:28.611063    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:28.612177 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:28.612154    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:24:29.159411 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.159391    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:24:29.311586 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.311557    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz5cv\" (UniqueName: \"kubernetes.io/projected/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-api-access-rz5cv\") pod \"0489a9a6-a24c-415c-b33a-9174a389c8f1\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") "
Jul 09 18:24:29.311732 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.311595    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-rbac-proxy-sar-config\") pod \"0489a9a6-a24c-415c-b33a-9174a389c8f1\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") "
Jul 09 18:24:29.311732 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.311615    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0489a9a6-a24c-415c-b33a-9174a389c8f1-proxy-tls\") pod \"0489a9a6-a24c-415c-b33a-9174a389c8f1\" (UID: \"0489a9a6-a24c-415c-b33a-9174a389c8f1\") "
Jul 09 18:24:29.311939 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.311916    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0489a9a6-a24c-415c-b33a-9174a389c8f1" (UID: "0489a9a6-a24c-415c-b33a-9174a389c8f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:29.313556 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.313533    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0489a9a6-a24c-415c-b33a-9174a389c8f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0489a9a6-a24c-415c-b33a-9174a389c8f1" (UID: "0489a9a6-a24c-415c-b33a-9174a389c8f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:29.313676 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.313658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-api-access-rz5cv" (OuterVolumeSpecName: "kube-api-access-rz5cv") pod "0489a9a6-a24c-415c-b33a-9174a389c8f1" (UID: "0489a9a6-a24c-415c-b33a-9174a389c8f1"). InnerVolumeSpecName "kube-api-access-rz5cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:29.412652 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.412626    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz5cv\" (UniqueName: \"kubernetes.io/projected/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-api-access-rz5cv\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:29.412652 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.412648    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0489a9a6-a24c-415c-b33a-9174a389c8f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:29.412776 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.412658    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0489a9a6-a24c-415c-b33a-9174a389c8f1-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:29.616231 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.616171    2560 generic.go:356] "Generic (PLEG): container finished" podID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerID="3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea" exitCode=0
Jul 09 18:24:29.616590 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.616230    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" event={"ID":"0489a9a6-a24c-415c-b33a-9174a389c8f1","Type":"ContainerDied","Data":"3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea"}
Jul 09 18:24:29.616590 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.616283    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9" event={"ID":"0489a9a6-a24c-415c-b33a-9174a389c8f1","Type":"ContainerDied","Data":"092d2c4cb73ff3ac0aa88713ee886b7a743e3560ae156896d20f04de201e2ed7"}
Jul 09 18:24:29.616590 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.616308    2560 scope.go:117] "RemoveContainer" containerID="2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53"
Jul 09 18:24:29.616590 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.616236    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"
Jul 09 18:24:29.616814 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.616644    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:24:29.625461 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.625445    2560 scope.go:117] "RemoveContainer" containerID="3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea"
Jul 09 18:24:29.632714 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.632699    2560 scope.go:117] "RemoveContainer" containerID="2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53"
Jul 09 18:24:29.632944 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:24:29.632926    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53\": container with ID starting with 2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53 not found: ID does not exist" containerID="2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53"
Jul 09 18:24:29.632991 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.632952    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53"} err="failed to get container status \"2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53\": rpc error: code = NotFound desc = could not find container \"2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53\": container with ID starting with 2b05e93a89af87c77ea061329753e012b3c9be0c2446215ee4da99cd7f645d53 not found: ID does not exist"
Jul 09 18:24:29.632991 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.632968    2560 scope.go:117] "RemoveContainer" containerID="3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea"
Jul 09 18:24:29.633171 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:24:29.633155    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea\": container with ID starting with 3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea not found: ID does not exist" containerID="3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea"
Jul 09 18:24:29.633209 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.633177    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea"} err="failed to get container status \"3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea\": rpc error: code = NotFound desc = could not find container \"3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea\": container with ID starting with 3c1630d8ce0478c44a259dd16d19342fe73807f9b5793a1e7754c4a9a7bf86ea not found: ID does not exist"
Jul 09 18:24:29.639229 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.639209    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"]
Jul 09 18:24:29.649253 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:29.649231    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea8a9-predictor-74877969-qprj9"]
Jul 09 18:24:30.390186 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:30.390145    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" path="/var/lib/kubelet/pods/0489a9a6-a24c-415c-b33a-9174a389c8f1/volumes"
Jul 09 18:24:34.625925 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:34.625894    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:24:34.626430 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:34.626364    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:24:44.626821 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:44.626773    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:24:54.626711 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:54.626668    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:24:56.005242 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.005202    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:56.068582 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.068554    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"]
Jul 09 18:24:56.068799 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.068773    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container" containerID="cri-o://625b66987444c8341eea03d4780e3bea9e9bd05b85e19f773a428eef204edaa5" gracePeriod=30
Jul 09 18:24:56.068943 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.068809    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kube-rbac-proxy" containerID="cri-o://f9c239b1061911344ee06a85de9ecdc38449bec7b750b3cdd4987a97a4c23334" gracePeriod=30
Jul 09 18:24:56.096712 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.096687    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:56.183585 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.183409    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:56.293943 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.293870    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:56.302551 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.302522    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"]
Jul 09 18:24:56.302913 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.302897    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kube-rbac-proxy"
Jul 09 18:24:56.302913 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.302917    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kube-rbac-proxy"
Jul 09 18:24:56.303005 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.302939    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container"
Jul 09 18:24:56.303005 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.302945    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container"
Jul 09 18:24:56.303005 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.302998    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kserve-container"
Jul 09 18:24:56.303105 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.303007    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0489a9a6-a24c-415c-b33a-9174a389c8f1" containerName="kube-rbac-proxy"
Jul 09 18:24:56.305972 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.305957    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.311706 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.311682    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e68f-predictor-serving-cert\""
Jul 09 18:24:56.311798 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.311735    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e68f-kube-rbac-proxy-sar-config\""
Jul 09 18:24:56.318387 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.318369    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"]
Jul 09 18:24:56.391594 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.391551    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:56.398652 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.398630    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.398765 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.398679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mw8\" (UniqueName: \"kubernetes.io/projected/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-api-access-66mw8\") pod \"error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.398765 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.398707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9599298c-1d64-4cf0-9624-91c3ff3b666a-proxy-tls\") pod \"error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.499657 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.499625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.499792 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.499692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mw8\" (UniqueName: \"kubernetes.io/projected/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-api-access-66mw8\") pod \"error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.499868 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.499790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9599298c-1d64-4cf0-9624-91c3ff3b666a-proxy-tls\") pod \"error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.500358 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.500331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.500757 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.500733    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:56.502524 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.502501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9599298c-1d64-4cf0-9624-91c3ff3b666a-proxy-tls\") pod \"error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.509181 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.509152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mw8\" (UniqueName: \"kubernetes.io/projected/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-api-access-66mw8\") pod \"error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.616362 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.616306    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:56.738850 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.738824    2560 generic.go:356] "Generic (PLEG): container finished" podID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerID="f9c239b1061911344ee06a85de9ecdc38449bec7b750b3cdd4987a97a4c23334" exitCode=2
Jul 09 18:24:56.738963 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.738892    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" event={"ID":"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2","Type":"ContainerDied","Data":"f9c239b1061911344ee06a85de9ecdc38449bec7b750b3cdd4987a97a4c23334"}
Jul 09 18:24:56.746182 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:56.746161    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"]
Jul 09 18:24:56.748763 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:24:56.748742    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9599298c_1d64_4cf0_9624_91c3ff3b666a.slice/crio-3b32769e69bf457d93ab878bba27ecb73072380804bb8c798513be3b9194d904 WatchSource:0}: Error finding container 3b32769e69bf457d93ab878bba27ecb73072380804bb8c798513be3b9194d904: Status 404 returned error can't find the container with id 3b32769e69bf457d93ab878bba27ecb73072380804bb8c798513be3b9194d904
Jul 09 18:24:57.745422 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:57.745388    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" event={"ID":"9599298c-1d64-4cf0-9624-91c3ff3b666a","Type":"ContainerStarted","Data":"0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1"}
Jul 09 18:24:57.745907 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:57.745447    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:57.745907 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:57.745470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" event={"ID":"9599298c-1d64-4cf0-9624-91c3ff3b666a","Type":"ContainerStarted","Data":"893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac"}
Jul 09 18:24:57.745907 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:57.745605    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:24:57.745907 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:57.745647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" event={"ID":"9599298c-1d64-4cf0-9624-91c3ff3b666a","Type":"ContainerStarted","Data":"3b32769e69bf457d93ab878bba27ecb73072380804bb8c798513be3b9194d904"}
Jul 09 18:24:57.746650 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:57.746629    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 09 18:24:57.765086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:57.765046    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podStartSLOduration=1.7650326299999999 podStartE2EDuration="1.76503263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:57.763426098 +0000 UTC m=+1241.915063425" watchObservedRunningTime="2026-07-09 18:24:57.76503263 +0000 UTC m=+1241.916669904"
Jul 09 18:24:58.752392 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.752360    2560 generic.go:356] "Generic (PLEG): container finished" podID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerID="625b66987444c8341eea03d4780e3bea9e9bd05b85e19f773a428eef204edaa5" exitCode=0
Jul 09 18:24:58.752768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.752432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" event={"ID":"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2","Type":"ContainerDied","Data":"625b66987444c8341eea03d4780e3bea9e9bd05b85e19f773a428eef204edaa5"}
Jul 09 18:24:58.752768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.752747    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 09 18:24:58.806264 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.806243    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:24:58.918195 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.918108    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlf57\" (UniqueName: \"kubernetes.io/projected/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-api-access-jlf57\") pod \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") "
Jul 09 18:24:58.918195 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.918166    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-proxy-tls\") pod \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") "
Jul 09 18:24:58.918371 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.918259    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-rbac-proxy-sar-config\") pod \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\" (UID: \"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2\") "
Jul 09 18:24:58.918630 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.918607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" (UID: "52c2d7ec-855f-42a1-9c52-7dd4c2b720c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:58.920178 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.920151    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-api-access-jlf57" (OuterVolumeSpecName: "kube-api-access-jlf57") pod "52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" (UID: "52c2d7ec-855f-42a1-9c52-7dd4c2b720c2"). InnerVolumeSpecName "kube-api-access-jlf57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:58.920256 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:58.920161    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" (UID: "52c2d7ec-855f-42a1-9c52-7dd4c2b720c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:59.019357 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:59.019331    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:59.019357 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:59.019354    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlf57\" (UniqueName: \"kubernetes.io/projected/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-kube-api-access-jlf57\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:59.019495 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:59.019363    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:59.758859 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:59.758831    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"
Jul 09 18:24:59.759351 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:59.758830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd" event={"ID":"52c2d7ec-855f-42a1-9c52-7dd4c2b720c2","Type":"ContainerDied","Data":"bbe6620ae84e23170a1f69fc5e06f0f2900fed5ff2a721108df0351f75d8ce55"}
Jul 09 18:24:59.759351 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:59.758965    2560 scope.go:117] "RemoveContainer" containerID="f9c239b1061911344ee06a85de9ecdc38449bec7b750b3cdd4987a97a4c23334"
Jul 09 18:24:59.768411 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:59.768392    2560 scope.go:117] "RemoveContainer" containerID="625b66987444c8341eea03d4780e3bea9e9bd05b85e19f773a428eef204edaa5"
Jul 09 18:24:59.788794 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:59.788764    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"]
Jul 09 18:24:59.790788 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:24:59.790762    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-04bd4-predictor-5bf6d6f9fc-74sxd"]
Jul 09 18:25:00.389191 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:00.389156    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" path="/var/lib/kubelet/pods/52c2d7ec-855f-42a1-9c52-7dd4c2b720c2/volumes"
Jul 09 18:25:03.757142 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:03.757114    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:25:03.757588 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:03.757558    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 09 18:25:04.627126 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:04.627087    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 09 18:25:13.758117 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:13.758039    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 09 18:25:14.628239 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:14.628209    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:25:23.758390 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:23.758345    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 09 18:25:26.613579 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:26.613547    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:26.732747 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:26.732723    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:33.757566 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:33.757529    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 09 18:25:36.553187 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.553157    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:36.626009 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.625981    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"]
Jul 09 18:25:36.626220 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.626194    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container" containerID="cri-o://23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a" gracePeriod=30
Jul 09 18:25:36.626290 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.626229    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kube-rbac-proxy" containerID="cri-o://6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f" gracePeriod=30
Jul 09 18:25:36.649412 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.649386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:36.734172 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.734129    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:36.805506 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.805401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:36.882242 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.882217    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"]
Jul 09 18:25:36.882624 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.882612    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container"
Jul 09 18:25:36.882674 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.882627    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container"
Jul 09 18:25:36.882674 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.882642    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kube-rbac-proxy"
Jul 09 18:25:36.882674 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.882648    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kube-rbac-proxy"
Jul 09 18:25:36.882767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.882698    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kube-rbac-proxy"
Jul 09 18:25:36.882767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.882711    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c2d7ec-855f-42a1-9c52-7dd4c2b720c2" containerName="kserve-container"
Jul 09 18:25:36.885697 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.885678    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:36.888331 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.888301    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fdee6-predictor-serving-cert\""
Jul 09 18:25:36.888435 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.888302    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fdee6-kube-rbac-proxy-sar-config\""
Jul 09 18:25:36.899256 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.899233    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"]
Jul 09 18:25:36.921049 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.921019    2560 generic.go:356] "Generic (PLEG): container finished" podID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerID="6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f" exitCode=2
Jul 09 18:25:36.921151 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.921092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" event={"ID":"ef7921fc-9596-4ec7-a2d2-da1101506d46","Type":"ContainerDied","Data":"6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f"}
Jul 09 18:25:36.960346 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.960306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:36.998588 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.998561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12feab4-b951-4bc4-812c-d953506ab50f-proxy-tls\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:36.998693 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.998609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12feab4-b951-4bc4-812c-d953506ab50f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:36.998693 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:36.998650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rlkd\" (UniqueName: \"kubernetes.io/projected/d12feab4-b951-4bc4-812c-d953506ab50f-kube-api-access-5rlkd\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:37.100022 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.099954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12feab4-b951-4bc4-812c-d953506ab50f-proxy-tls\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:37.100022 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.099993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12feab4-b951-4bc4-812c-d953506ab50f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:37.100234 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:25:37.100111    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-serving-cert: secret "error-404-isvc-fdee6-predictor-serving-cert" not found
Jul 09 18:25:37.100234 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.100120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlkd\" (UniqueName: \"kubernetes.io/projected/d12feab4-b951-4bc4-812c-d953506ab50f-kube-api-access-5rlkd\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:37.100234 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:25:37.100187    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12feab4-b951-4bc4-812c-d953506ab50f-proxy-tls podName:d12feab4-b951-4bc4-812c-d953506ab50f nodeName:}" failed. No retries permitted until 2026-07-09 18:25:37.600165739 +0000 UTC m=+1281.751803004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d12feab4-b951-4bc4-812c-d953506ab50f-proxy-tls") pod "error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" (UID: "d12feab4-b951-4bc4-812c-d953506ab50f") : secret "error-404-isvc-fdee6-predictor-serving-cert" not found
Jul 09 18:25:37.100681 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.100664    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12feab4-b951-4bc4-812c-d953506ab50f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:37.121986 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.121962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rlkd\" (UniqueName: \"kubernetes.io/projected/d12feab4-b951-4bc4-812c-d953506ab50f-kube-api-access-5rlkd\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:37.361388 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.361318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:37.603986 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.603954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12feab4-b951-4bc4-812c-d953506ab50f-proxy-tls\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:37.606369 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.606347    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12feab4-b951-4bc4-812c-d953506ab50f-proxy-tls\") pod \"error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") " pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:37.795934 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.795890    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:37.927685 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:37.927665    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"]
Jul 09 18:25:37.929414 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:25:37.929389    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12feab4_b951_4bc4_812c_d953506ab50f.slice/crio-456b34ec51fb0720fd3ce3cbf4496719e502a2939574e93931796fbc91340ed9 WatchSource:0}: Error finding container 456b34ec51fb0720fd3ce3cbf4496719e502a2939574e93931796fbc91340ed9: Status 404 returned error can't find the container with id 456b34ec51fb0720fd3ce3cbf4496719e502a2939574e93931796fbc91340ed9
Jul 09 18:25:38.933451 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:38.933416    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" event={"ID":"d12feab4-b951-4bc4-812c-d953506ab50f","Type":"ContainerStarted","Data":"b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9"}
Jul 09 18:25:38.933883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:38.933498    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" event={"ID":"d12feab4-b951-4bc4-812c-d953506ab50f","Type":"ContainerStarted","Data":"46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231"}
Jul 09 18:25:38.933883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:38.933650    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:38.933883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:38.933713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" event={"ID":"d12feab4-b951-4bc4-812c-d953506ab50f","Type":"ContainerStarted","Data":"456b34ec51fb0720fd3ce3cbf4496719e502a2939574e93931796fbc91340ed9"}
Jul 09 18:25:38.933883 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:38.933741    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:38.935017 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:38.934994    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:25:38.957679 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:38.957635    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podStartSLOduration=2.957621806 podStartE2EDuration="2.957621806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:38.957153833 +0000 UTC m=+1283.108791126" watchObservedRunningTime="2026-07-09 18:25:38.957621806 +0000 UTC m=+1283.109259078"
Jul 09 18:25:39.616871 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:39.616834    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 09 18:25:39.938071 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:39.938038    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:25:40.078557 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.078535    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:25:40.226342 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.226274    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-rbac-proxy-sar-config\") pod \"ef7921fc-9596-4ec7-a2d2-da1101506d46\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") "
Jul 09 18:25:40.226342 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.226335    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wxjx\" (UniqueName: \"kubernetes.io/projected/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-api-access-2wxjx\") pod \"ef7921fc-9596-4ec7-a2d2-da1101506d46\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") "
Jul 09 18:25:40.226554 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.226376    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7921fc-9596-4ec7-a2d2-da1101506d46-proxy-tls\") pod \"ef7921fc-9596-4ec7-a2d2-da1101506d46\" (UID: \"ef7921fc-9596-4ec7-a2d2-da1101506d46\") "
Jul 09 18:25:40.226721 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.226690    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef7921fc-9596-4ec7-a2d2-da1101506d46" (UID: "ef7921fc-9596-4ec7-a2d2-da1101506d46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:25:40.228461 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.228430    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-api-access-2wxjx" (OuterVolumeSpecName: "kube-api-access-2wxjx") pod "ef7921fc-9596-4ec7-a2d2-da1101506d46" (UID: "ef7921fc-9596-4ec7-a2d2-da1101506d46"). InnerVolumeSpecName "kube-api-access-2wxjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:25:40.228461 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.228442    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7921fc-9596-4ec7-a2d2-da1101506d46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef7921fc-9596-4ec7-a2d2-da1101506d46" (UID: "ef7921fc-9596-4ec7-a2d2-da1101506d46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:25:40.326950 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.326913    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7921fc-9596-4ec7-a2d2-da1101506d46-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:25:40.327101 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.326963    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:25:40.327101 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.326973    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wxjx\" (UniqueName: \"kubernetes.io/projected/ef7921fc-9596-4ec7-a2d2-da1101506d46-kube-api-access-2wxjx\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:25:40.943944 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.943902    2560 generic.go:356] "Generic (PLEG): container finished" podID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerID="23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a" exitCode=0
Jul 09 18:25:40.944383 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.943955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" event={"ID":"ef7921fc-9596-4ec7-a2d2-da1101506d46","Type":"ContainerDied","Data":"23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a"}
Jul 09 18:25:40.944383 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.943978    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"
Jul 09 18:25:40.944383 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.944019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n" event={"ID":"ef7921fc-9596-4ec7-a2d2-da1101506d46","Type":"ContainerDied","Data":"fe6304861e0d72be595c0e8093466fcf4f57686131defd84ce131198d4386288"}
Jul 09 18:25:40.944383 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.944054    2560 scope.go:117] "RemoveContainer" containerID="6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f"
Jul 09 18:25:40.951992 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.951975    2560 scope.go:117] "RemoveContainer" containerID="23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a"
Jul 09 18:25:40.958915 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.958895    2560 scope.go:117] "RemoveContainer" containerID="6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f"
Jul 09 18:25:40.959150 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:25:40.959130    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f\": container with ID starting with 6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f not found: ID does not exist" containerID="6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f"
Jul 09 18:25:40.959195 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.959157    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f"} err="failed to get container status \"6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f\": rpc error: code = NotFound desc = could not find container \"6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f\": container with ID starting with 6caeb02e82708e3301b8914895a196fcfd30e083abdefa22e102e43f8896f45f not found: ID does not exist"
Jul 09 18:25:40.959195 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.959173    2560 scope.go:117] "RemoveContainer" containerID="23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a"
Jul 09 18:25:40.959378 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:25:40.959360    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a\": container with ID starting with 23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a not found: ID does not exist" containerID="23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a"
Jul 09 18:25:40.959418 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.959384    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a"} err="failed to get container status \"23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a\": rpc error: code = NotFound desc = could not find container \"23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a\": container with ID starting with 23309d6282b6c116093e4e9abb4b647ee9a7ae3b057750385ca241473d36fc5a not found: ID does not exist"
Jul 09 18:25:40.961728 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.961709    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"]
Jul 09 18:25:40.965843 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:40.965823    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3ac8-predictor-59486dd895-2dj4n"]
Jul 09 18:25:42.389472 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:42.389441    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" path="/var/lib/kubelet/pods/ef7921fc-9596-4ec7-a2d2-da1101506d46/volumes"
Jul 09 18:25:43.757952 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:43.757931    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:25:44.942411 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:44.942387    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:25:44.942881 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:44.942857    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:25:54.943168 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:54.943132    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:25:56.213551 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:56.213510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:25:57.720607 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:25:57.720576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:04.943016 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:04.942977    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:26:06.334341 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.334304    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:06.504997 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.504962    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:06.520236 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.520210    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"]
Jul 09 18:26:06.520444 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.520408    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container" containerID="cri-o://893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac" gracePeriod=30
Jul 09 18:26:06.520599 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.520456    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kube-rbac-proxy" containerID="cri-o://0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1" gracePeriod=30
Jul 09 18:26:06.596324 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.596249    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:06.625421 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.625391    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"]
Jul 09 18:26:06.625830 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.625812    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container"
Jul 09 18:26:06.625920 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.625832    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container"
Jul 09 18:26:06.625920 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.625892    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kube-rbac-proxy"
Jul 09 18:26:06.625920 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.625903    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kube-rbac-proxy"
Jul 09 18:26:06.626078 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.625978    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kube-rbac-proxy"
Jul 09 18:26:06.626078 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.625996    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7921fc-9596-4ec7-a2d2-da1101506d46" containerName="kserve-container"
Jul 09 18:26:06.629255 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.629237    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.631722 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.631705    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4ab35-predictor-serving-cert\""
Jul 09 18:26:06.631832 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.631812    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4ab35-kube-rbac-proxy-sar-config\""
Jul 09 18:26:06.642701 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.642679    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"]
Jul 09 18:26:06.669904 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.669878    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:06.735755 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.735733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b014cfd-b839-4e4a-912f-81a85924b6bf-proxy-tls\") pod \"error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") " pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.735879 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.735782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") " pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.735879 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.735813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrcr\" (UniqueName: \"kubernetes.io/projected/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-api-access-wnrcr\") pod \"error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") " pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.785506 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.785455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:06.836591 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.836565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrcr\" (UniqueName: \"kubernetes.io/projected/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-api-access-wnrcr\") pod \"error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") " pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.836702 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.836607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b014cfd-b839-4e4a-912f-81a85924b6bf-proxy-tls\") pod \"error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") " pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.836702 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.836645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") " pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.837332 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.837312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") " pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.839017 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.838996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b014cfd-b839-4e4a-912f-81a85924b6bf-proxy-tls\") pod \"error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") " pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.844973 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.844951    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrcr\" (UniqueName: \"kubernetes.io/projected/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-api-access-wnrcr\") pod \"error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") " pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:06.940569 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:06.940513    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:07.061840 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:07.061810    2560 generic.go:356] "Generic (PLEG): container finished" podID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerID="0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1" exitCode=2
Jul 09 18:26:07.062001 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:07.061877    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" event={"ID":"9599298c-1d64-4cf0-9624-91c3ff3b666a","Type":"ContainerDied","Data":"0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1"}
Jul 09 18:26:07.068755 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:07.068734    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"]
Jul 09 18:26:07.070867 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:26:07.070847    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b014cfd_b839_4e4a_912f_81a85924b6bf.slice/crio-31837845dfe43a0ba4285b5bc16efd38dfe9b5f28ce1085cf000094d8675d2d3 WatchSource:0}: Error finding container 31837845dfe43a0ba4285b5bc16efd38dfe9b5f28ce1085cf000094d8675d2d3: Status 404 returned error can't find the container with id 31837845dfe43a0ba4285b5bc16efd38dfe9b5f28ce1085cf000094d8675d2d3
Jul 09 18:26:08.068397 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:08.068364    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" event={"ID":"6b014cfd-b839-4e4a-912f-81a85924b6bf","Type":"ContainerStarted","Data":"e47ee275fe0ec3721ff7cec869012fd14ba130b20968e1baf2b4ee64d9af9aea"}
Jul 09 18:26:08.068790 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:08.068418    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" event={"ID":"6b014cfd-b839-4e4a-912f-81a85924b6bf","Type":"ContainerStarted","Data":"8b69e3640b9d59c439282f548babca131a12e22f7688f320ae3ef5850df1a774"}
Jul 09 18:26:08.068790 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:08.068436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" event={"ID":"6b014cfd-b839-4e4a-912f-81a85924b6bf","Type":"ContainerStarted","Data":"31837845dfe43a0ba4285b5bc16efd38dfe9b5f28ce1085cf000094d8675d2d3"}
Jul 09 18:26:08.068790 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:08.068460    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:08.068790 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:08.068514    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:08.069960 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:08.069934    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:26:08.091309 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:08.091236    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podStartSLOduration=2.091217218 podStartE2EDuration="2.091217218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:08.09029675 +0000 UTC m=+1312.241934022" watchObservedRunningTime="2026-07-09 18:26:08.091217218 +0000 UTC m=+1312.242854490"
Jul 09 18:26:08.753100 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:08.753065    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 09 18:26:09.072344 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.072246    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:26:09.269083 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.269059    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:26:09.358348 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.358286    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-rbac-proxy-sar-config\") pod \"9599298c-1d64-4cf0-9624-91c3ff3b666a\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") "
Jul 09 18:26:09.358470 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.358381    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66mw8\" (UniqueName: \"kubernetes.io/projected/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-api-access-66mw8\") pod \"9599298c-1d64-4cf0-9624-91c3ff3b666a\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") "
Jul 09 18:26:09.358470 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.358413    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9599298c-1d64-4cf0-9624-91c3ff3b666a-proxy-tls\") pod \"9599298c-1d64-4cf0-9624-91c3ff3b666a\" (UID: \"9599298c-1d64-4cf0-9624-91c3ff3b666a\") "
Jul 09 18:26:09.358625 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.358585    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9599298c-1d64-4cf0-9624-91c3ff3b666a" (UID: "9599298c-1d64-4cf0-9624-91c3ff3b666a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:09.360361 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.360336    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9599298c-1d64-4cf0-9624-91c3ff3b666a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9599298c-1d64-4cf0-9624-91c3ff3b666a" (UID: "9599298c-1d64-4cf0-9624-91c3ff3b666a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:09.360451 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.360384    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-api-access-66mw8" (OuterVolumeSpecName: "kube-api-access-66mw8") pod "9599298c-1d64-4cf0-9624-91c3ff3b666a" (UID: "9599298c-1d64-4cf0-9624-91c3ff3b666a"). InnerVolumeSpecName "kube-api-access-66mw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:09.459417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.459383    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:09.459417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.459414    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66mw8\" (UniqueName: \"kubernetes.io/projected/9599298c-1d64-4cf0-9624-91c3ff3b666a-kube-api-access-66mw8\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:09.459417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:09.459423    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9599298c-1d64-4cf0-9624-91c3ff3b666a-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:10.077711 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.077675    2560 generic.go:356] "Generic (PLEG): container finished" podID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerID="893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac" exitCode=0
Jul 09 18:26:10.078071 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.077724    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"
Jul 09 18:26:10.078071 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.077758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" event={"ID":"9599298c-1d64-4cf0-9624-91c3ff3b666a","Type":"ContainerDied","Data":"893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac"}
Jul 09 18:26:10.078071 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.077816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk" event={"ID":"9599298c-1d64-4cf0-9624-91c3ff3b666a","Type":"ContainerDied","Data":"3b32769e69bf457d93ab878bba27ecb73072380804bb8c798513be3b9194d904"}
Jul 09 18:26:10.078071 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.077846    2560 scope.go:117] "RemoveContainer" containerID="0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1"
Jul 09 18:26:10.087096 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.087072    2560 scope.go:117] "RemoveContainer" containerID="893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac"
Jul 09 18:26:10.094704 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.094687    2560 scope.go:117] "RemoveContainer" containerID="0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1"
Jul 09 18:26:10.094959 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:26:10.094932    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1\": container with ID starting with 0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1 not found: ID does not exist" containerID="0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1"
Jul 09 18:26:10.095004 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.094962    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1"} err="failed to get container status \"0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1\": rpc error: code = NotFound desc = could not find container \"0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1\": container with ID starting with 0624b4ea2860a99d201fef8541f2e34dba4a58f7a0ea7c20f70076551c2f1af1 not found: ID does not exist"
Jul 09 18:26:10.095004 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.094986    2560 scope.go:117] "RemoveContainer" containerID="893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac"
Jul 09 18:26:10.095228 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:26:10.095209    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac\": container with ID starting with 893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac not found: ID does not exist" containerID="893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac"
Jul 09 18:26:10.095288 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.095236    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac"} err="failed to get container status \"893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac\": rpc error: code = NotFound desc = could not find container \"893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac\": container with ID starting with 893dc33c15e08ac934d31d0f0798ab768524b54cd6cf460fd1eb202f31346aac not found: ID does not exist"
Jul 09 18:26:10.100340 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.100320    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"]
Jul 09 18:26:10.112319 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.112274    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e68f-predictor-59fc976fc5-4bbnk"]
Jul 09 18:26:10.390503 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:10.390418    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" path="/var/lib/kubelet/pods/9599298c-1d64-4cf0-9624-91c3ff3b666a/volumes"
Jul 09 18:26:14.077079 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:14.077049    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:26:14.077599 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:14.077574    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:26:14.943782 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:14.943740    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 09 18:26:24.078269 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:24.078228    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:26:24.943774 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:24.943746    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:26:34.078532 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:34.078497    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:26:36.757316 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:36.757283    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:36.981695 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:36.981661    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:44.077710 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:44.077615    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:26:46.139854 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:46.139818    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:54.079383 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:26:54.079347    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:27:06.586540 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:27:06.586505    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:27:06.662215 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:27:06.662179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:03.630855 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:28:03.630795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:16.339166 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:29:16.339136    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:29:16.343779 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:29:16.343759    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:32:22.215621 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:32:22.215587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:16.371545 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:16.371520    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:34:16.377526 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:16.377507    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:34:51.422050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.421988    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:51.601629 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.601593    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:51.753471 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.753432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:51.754408 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.754386    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"]
Jul 09 18:34:51.754764 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.754751    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container"
Jul 09 18:34:51.754810 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.754766    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container"
Jul 09 18:34:51.754810 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.754781    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kube-rbac-proxy"
Jul 09 18:34:51.754810 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.754786    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kube-rbac-proxy"
Jul 09 18:34:51.754907 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.754834    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kube-rbac-proxy"
Jul 09 18:34:51.754907 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.754846    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9599298c-1d64-4cf0-9624-91c3ff3b666a" containerName="kserve-container"
Jul 09 18:34:51.757858 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.757841    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:51.764990 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.764970    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-05680-predictor-serving-cert\""
Jul 09 18:34:51.765110 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.765010    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-05680-kube-rbac-proxy-sar-config\""
Jul 09 18:34:51.775807 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.775784    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"]
Jul 09 18:34:51.821971 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.821937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:51.821971 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.821978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-proxy-tls\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:51.822187 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.821995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8zq\" (UniqueName: \"kubernetes.io/projected/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-api-access-lw8zq\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:51.838131 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.838106    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"]
Jul 09 18:34:51.838339 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.838316    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container" containerID="cri-o://46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231" gracePeriod=30
Jul 09 18:34:51.838435 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.838402    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kube-rbac-proxy" containerID="cri-o://b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9" gracePeriod=30
Jul 09 18:34:51.876580 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.876550    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:51.923249 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.923221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-proxy-tls\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:51.923371 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.923252    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8zq\" (UniqueName: \"kubernetes.io/projected/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-api-access-lw8zq\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:51.923371 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.923329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:51.923465 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:34:51.923366    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-05680-predictor-serving-cert: secret "error-404-isvc-05680-predictor-serving-cert" not found
Jul 09 18:34:51.923465 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:34:51.923435    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-proxy-tls podName:66c6fbe6-a35e-4d92-b931-0f7c56aaa15e nodeName:}" failed. No retries permitted until 2026-07-09 18:34:52.423413929 +0000 UTC m=+1836.575051183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-proxy-tls") pod "error-404-isvc-05680-predictor-64bf589d5f-hrsh6" (UID: "66c6fbe6-a35e-4d92-b931-0f7c56aaa15e") : secret "error-404-isvc-05680-predictor-serving-cert" not found
Jul 09 18:34:51.924014 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.923998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:51.933473 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.933453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8zq\" (UniqueName: \"kubernetes.io/projected/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-api-access-lw8zq\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:51.997925 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:51.997891    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:52.277974 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:52.277927    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:34:52.374734 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:52.374704    2560 generic.go:356] "Generic (PLEG): container finished" podID="d12feab4-b951-4bc4-812c-d953506ab50f" containerID="b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9" exitCode=2
Jul 09 18:34:52.374900 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:52.374742    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" event={"ID":"d12feab4-b951-4bc4-812c-d953506ab50f","Type":"ContainerDied","Data":"b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9"}
Jul 09 18:34:52.426941 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:52.426909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-proxy-tls\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:52.429206 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:52.429188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-proxy-tls\") pod \"error-404-isvc-05680-predictor-64bf589d5f-hrsh6\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") " pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:52.668233 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:52.668157    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:52.802185 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:52.802156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"]
Jul 09 18:34:52.805411 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:34:52.805386    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c6fbe6_a35e_4d92_b931_0f7c56aaa15e.slice/crio-bc6e500f81922cdbdcac791cbc4bfecf28c5a66d8054afa821a3636eba0b9850 WatchSource:0}: Error finding container bc6e500f81922cdbdcac791cbc4bfecf28c5a66d8054afa821a3636eba0b9850: Status 404 returned error can't find the container with id bc6e500f81922cdbdcac791cbc4bfecf28c5a66d8054afa821a3636eba0b9850
Jul 09 18:34:53.381239 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:53.381188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" event={"ID":"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e","Type":"ContainerStarted","Data":"8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae"}
Jul 09 18:34:53.381454 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:53.381285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" event={"ID":"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e","Type":"ContainerStarted","Data":"f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374"}
Jul 09 18:34:53.381454 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:53.381306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" event={"ID":"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e","Type":"ContainerStarted","Data":"bc6e500f81922cdbdcac791cbc4bfecf28c5a66d8054afa821a3636eba0b9850"}
Jul 09 18:34:53.381454 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:53.381327    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:53.412022 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:53.411966    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" podStartSLOduration=2.4119514730000002 podStartE2EDuration="2.411951473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:53.409208111 +0000 UTC m=+1837.560845382" watchObservedRunningTime="2026-07-09 18:34:53.411951473 +0000 UTC m=+1837.563588745"
Jul 09 18:34:54.388767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:54.388729    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:34:54.389767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:54.389744    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:34:54.993755 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:54.993733    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:34:55.050968 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.050903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12feab4-b951-4bc4-812c-d953506ab50f-kube-rbac-proxy-sar-config\") pod \"d12feab4-b951-4bc4-812c-d953506ab50f\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") "
Jul 09 18:34:55.050968 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.050958    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12feab4-b951-4bc4-812c-d953506ab50f-proxy-tls\") pod \"d12feab4-b951-4bc4-812c-d953506ab50f\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") "
Jul 09 18:34:55.051117 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.051009    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rlkd\" (UniqueName: \"kubernetes.io/projected/d12feab4-b951-4bc4-812c-d953506ab50f-kube-api-access-5rlkd\") pod \"d12feab4-b951-4bc4-812c-d953506ab50f\" (UID: \"d12feab4-b951-4bc4-812c-d953506ab50f\") "
Jul 09 18:34:55.051295 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.051264    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12feab4-b951-4bc4-812c-d953506ab50f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d12feab4-b951-4bc4-812c-d953506ab50f" (UID: "d12feab4-b951-4bc4-812c-d953506ab50f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:34:55.053003 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.052975    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12feab4-b951-4bc4-812c-d953506ab50f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d12feab4-b951-4bc4-812c-d953506ab50f" (UID: "d12feab4-b951-4bc4-812c-d953506ab50f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:34:55.053086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.053002    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12feab4-b951-4bc4-812c-d953506ab50f-kube-api-access-5rlkd" (OuterVolumeSpecName: "kube-api-access-5rlkd") pod "d12feab4-b951-4bc4-812c-d953506ab50f" (UID: "d12feab4-b951-4bc4-812c-d953506ab50f"). InnerVolumeSpecName "kube-api-access-5rlkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:34:55.151735 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.151707    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12feab4-b951-4bc4-812c-d953506ab50f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:34:55.151735 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.151734    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12feab4-b951-4bc4-812c-d953506ab50f-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:34:55.151895 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.151753    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rlkd\" (UniqueName: \"kubernetes.io/projected/d12feab4-b951-4bc4-812c-d953506ab50f-kube-api-access-5rlkd\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:34:55.393846 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.393740    2560 generic.go:356] "Generic (PLEG): container finished" podID="d12feab4-b951-4bc4-812c-d953506ab50f" containerID="46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231" exitCode=0
Jul 09 18:34:55.393846 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.393807    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"
Jul 09 18:34:55.393846 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.393824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" event={"ID":"d12feab4-b951-4bc4-812c-d953506ab50f","Type":"ContainerDied","Data":"46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231"}
Jul 09 18:34:55.394428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.393878    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" event={"ID":"d12feab4-b951-4bc4-812c-d953506ab50f","Type":"ContainerDied","Data":"456b34ec51fb0720fd3ce3cbf4496719e502a2939574e93931796fbc91340ed9"}
Jul 09 18:34:55.394428 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.393906    2560 scope.go:117] "RemoveContainer" containerID="b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9"
Jul 09 18:34:55.394580 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.394554    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:34:55.402910 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.402895    2560 scope.go:117] "RemoveContainer" containerID="46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231"
Jul 09 18:34:55.410165 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.410148    2560 scope.go:117] "RemoveContainer" containerID="b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9"
Jul 09 18:34:55.410408 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:34:55.410389    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9\": container with ID starting with b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9 not found: ID does not exist" containerID="b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9"
Jul 09 18:34:55.410456 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.410416    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9"} err="failed to get container status \"b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9\": rpc error: code = NotFound desc = could not find container \"b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9\": container with ID starting with b3cffa4254e59a99ffc3734d1354fbb9d8a6a701b8b4dcf31f6f7369572893c9 not found: ID does not exist"
Jul 09 18:34:55.410456 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.410433    2560 scope.go:117] "RemoveContainer" containerID="46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231"
Jul 09 18:34:55.410687 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:34:55.410669    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231\": container with ID starting with 46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231 not found: ID does not exist" containerID="46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231"
Jul 09 18:34:55.410723 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.410692    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231"} err="failed to get container status \"46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231\": rpc error: code = NotFound desc = could not find container \"46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231\": container with ID starting with 46ce6f59b404119177b1c4d1d9ce95664d3505dbb278d837daa11e674f8ff231 not found: ID does not exist"
Jul 09 18:34:55.422504 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.422469    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"]
Jul 09 18:34:55.426870 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.426848    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm"]
Jul 09 18:34:55.939366 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.939327    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": context deadline exceeded"
Jul 09 18:34:55.944259 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:55.944221    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fdee6-predictor-67f7db74c7-mf4lm" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: i/o timeout"
Jul 09 18:34:56.390645 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:34:56.390611    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" path="/var/lib/kubelet/pods/d12feab4-b951-4bc4-812c-d953506ab50f/volumes"
Jul 09 18:35:00.399033 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:00.399003    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:35:00.399538 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:00.399433    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:35:10.400004 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:10.399959    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:35:20.399709 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:20.399677    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:35:21.316578 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.316541    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:35:21.422326 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.422298    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"]
Jul 09 18:35:21.423106 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.423070    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container" containerID="cri-o://8b69e3640b9d59c439282f548babca131a12e22f7688f320ae3ef5850df1a774" gracePeriod=30
Jul 09 18:35:21.423359 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.423044    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kube-rbac-proxy" containerID="cri-o://e47ee275fe0ec3721ff7cec869012fd14ba130b20968e1baf2b4ee64d9af9aea" gracePeriod=30
Jul 09 18:35:21.476133 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.476098    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:35:21.618925 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.618846    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:35:21.721801 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.721767    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:35:21.734543 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.734518    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"]
Jul 09 18:35:21.734948 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.734933    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kube-rbac-proxy"
Jul 09 18:35:21.734992 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.734950    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kube-rbac-proxy"
Jul 09 18:35:21.734992 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.734974    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container"
Jul 09 18:35:21.734992 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.734980    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container"
Jul 09 18:35:21.735084 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.735032    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kube-rbac-proxy"
Jul 09 18:35:21.735084 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.735041    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12feab4-b951-4bc4-812c-d953506ab50f" containerName="kserve-container"
Jul 09 18:35:21.738053 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.738037    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:21.741003 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.740980    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-95b80-predictor-serving-cert\""
Jul 09 18:35:21.741102 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.741046    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-95b80-kube-rbac-proxy-sar-config\""
Jul 09 18:35:21.753714 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.753693    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"]
Jul 09 18:35:21.816012 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.815966    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:35:21.853041 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.853005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab903443-f510-4930-ab86-f308ef331e65-proxy-tls\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:21.853196 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.853052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab903443-f510-4930-ab86-f308ef331e65-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:21.853196 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.853127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvxf\" (UniqueName: \"kubernetes.io/projected/ab903443-f510-4930-ab86-f308ef331e65-kube-api-access-crvxf\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:21.954042 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.953956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvxf\" (UniqueName: \"kubernetes.io/projected/ab903443-f510-4930-ab86-f308ef331e65-kube-api-access-crvxf\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:21.954224 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.954043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab903443-f510-4930-ab86-f308ef331e65-proxy-tls\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:21.954224 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.954073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab903443-f510-4930-ab86-f308ef331e65-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:21.954224 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:35:21.954190    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-95b80-predictor-serving-cert: secret "error-404-isvc-95b80-predictor-serving-cert" not found
Jul 09 18:35:21.954461 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:35:21.954261    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab903443-f510-4930-ab86-f308ef331e65-proxy-tls podName:ab903443-f510-4930-ab86-f308ef331e65 nodeName:}" failed. No retries permitted until 2026-07-09 18:35:22.454245403 +0000 UTC m=+1866.605882656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ab903443-f510-4930-ab86-f308ef331e65-proxy-tls") pod "error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" (UID: "ab903443-f510-4930-ab86-f308ef331e65") : secret "error-404-isvc-95b80-predictor-serving-cert" not found
Jul 09 18:35:21.954809 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.954787    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab903443-f510-4930-ab86-f308ef331e65-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:21.965570 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:21.965551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvxf\" (UniqueName: \"kubernetes.io/projected/ab903443-f510-4930-ab86-f308ef331e65-kube-api-access-crvxf\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:22.213510 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:22.213415    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:35:22.458032 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:22.457995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab903443-f510-4930-ab86-f308ef331e65-proxy-tls\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:22.460450 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:22.460423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab903443-f510-4930-ab86-f308ef331e65-proxy-tls\") pod \"error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") " pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:22.516542 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:22.516511    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerID="e47ee275fe0ec3721ff7cec869012fd14ba130b20968e1baf2b4ee64d9af9aea" exitCode=2
Jul 09 18:35:22.516708 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:22.516557    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" event={"ID":"6b014cfd-b839-4e4a-912f-81a85924b6bf","Type":"ContainerDied","Data":"e47ee275fe0ec3721ff7cec869012fd14ba130b20968e1baf2b4ee64d9af9aea"}
Jul 09 18:35:22.648281 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:22.648249    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:22.780529 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:22.780505    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"]
Jul 09 18:35:22.782837 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:35:22.782793    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab903443_f510_4930_ab86_f308ef331e65.slice/crio-a2a3dbc97960db2a50dff7c2952fa2ac6d932c6fba73a733402ad9809a333625 WatchSource:0}: Error finding container a2a3dbc97960db2a50dff7c2952fa2ac6d932c6fba73a733402ad9809a333625: Status 404 returned error can't find the container with id a2a3dbc97960db2a50dff7c2952fa2ac6d932c6fba73a733402ad9809a333625
Jul 09 18:35:23.526960 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:23.526921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" event={"ID":"ab903443-f510-4930-ab86-f308ef331e65","Type":"ContainerStarted","Data":"60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5"}
Jul 09 18:35:23.527375 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:23.526981    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" event={"ID":"ab903443-f510-4930-ab86-f308ef331e65","Type":"ContainerStarted","Data":"38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a"}
Jul 09 18:35:23.527375 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:23.527005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" event={"ID":"ab903443-f510-4930-ab86-f308ef331e65","Type":"ContainerStarted","Data":"a2a3dbc97960db2a50dff7c2952fa2ac6d932c6fba73a733402ad9809a333625"}
Jul 09 18:35:23.527375 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:23.527167    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:23.527375 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:23.527224    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:23.528608 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:23.528581    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 09 18:35:24.073515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.073453    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 09 18:35:24.077779 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.077754    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 09 18:35:24.534907 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.534874    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerID="8b69e3640b9d59c439282f548babca131a12e22f7688f320ae3ef5850df1a774" exitCode=0
Jul 09 18:35:24.535287 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.534949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" event={"ID":"6b014cfd-b839-4e4a-912f-81a85924b6bf","Type":"ContainerDied","Data":"8b69e3640b9d59c439282f548babca131a12e22f7688f320ae3ef5850df1a774"}
Jul 09 18:35:24.535287 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.535262    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 09 18:35:24.570915 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.570892    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:35:24.595104 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.595056    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" podStartSLOduration=3.595038012 podStartE2EDuration="3.595038012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:35:23.570537268 +0000 UTC m=+1867.722174558" watchObservedRunningTime="2026-07-09 18:35:24.595038012 +0000 UTC m=+1868.746675335"
Jul 09 18:35:24.678460 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.678369    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnrcr\" (UniqueName: \"kubernetes.io/projected/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-api-access-wnrcr\") pod \"6b014cfd-b839-4e4a-912f-81a85924b6bf\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") "
Jul 09 18:35:24.678460 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.678418    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b014cfd-b839-4e4a-912f-81a85924b6bf-proxy-tls\") pod \"6b014cfd-b839-4e4a-912f-81a85924b6bf\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") "
Jul 09 18:35:24.678460 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.678462    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-rbac-proxy-sar-config\") pod \"6b014cfd-b839-4e4a-912f-81a85924b6bf\" (UID: \"6b014cfd-b839-4e4a-912f-81a85924b6bf\") "
Jul 09 18:35:24.678940 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.678905    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b014cfd-b839-4e4a-912f-81a85924b6bf" (UID: "6b014cfd-b839-4e4a-912f-81a85924b6bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:35:24.680542 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.680514    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b014cfd-b839-4e4a-912f-81a85924b6bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b014cfd-b839-4e4a-912f-81a85924b6bf" (UID: "6b014cfd-b839-4e4a-912f-81a85924b6bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:35:24.680674 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.680660    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-api-access-wnrcr" (OuterVolumeSpecName: "kube-api-access-wnrcr") pod "6b014cfd-b839-4e4a-912f-81a85924b6bf" (UID: "6b014cfd-b839-4e4a-912f-81a85924b6bf"). InnerVolumeSpecName "kube-api-access-wnrcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:35:24.779566 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.779528    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b014cfd-b839-4e4a-912f-81a85924b6bf-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:35:24.779566 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.779558    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:35:24.779566 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:24.779571    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnrcr\" (UniqueName: \"kubernetes.io/projected/6b014cfd-b839-4e4a-912f-81a85924b6bf-kube-api-access-wnrcr\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:35:25.541906 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:25.541876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57" event={"ID":"6b014cfd-b839-4e4a-912f-81a85924b6bf","Type":"ContainerDied","Data":"31837845dfe43a0ba4285b5bc16efd38dfe9b5f28ce1085cf000094d8675d2d3"}
Jul 09 18:35:25.541906 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:25.541908    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"
Jul 09 18:35:25.542408 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:25.541941    2560 scope.go:117] "RemoveContainer" containerID="e47ee275fe0ec3721ff7cec869012fd14ba130b20968e1baf2b4ee64d9af9aea"
Jul 09 18:35:25.550111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:25.550092    2560 scope.go:117] "RemoveContainer" containerID="8b69e3640b9d59c439282f548babca131a12e22f7688f320ae3ef5850df1a774"
Jul 09 18:35:25.570843 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:25.570816    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"]
Jul 09 18:35:25.576256 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:25.576232    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4ab35-predictor-5897c6f9ff-ghl57"]
Jul 09 18:35:26.390537 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:26.390503    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" path="/var/lib/kubelet/pods/6b014cfd-b839-4e4a-912f-81a85924b6bf/volumes"
Jul 09 18:35:29.539559 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:29.539529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:35:29.540077 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:29.540046    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 09 18:35:30.399786 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:30.399750    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 09 18:35:39.540730 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:39.540644    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 09 18:35:40.400586 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:40.400557    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:35:49.540905 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:49.540869    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 09 18:35:51.675866 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:51.675830    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:35:51.800539 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:51.800511    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:35:59.540114 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:35:59.540073    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 09 18:36:01.794994 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:01.794962    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:36:01.970968 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:01.970929    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:36:02.067415 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.067335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:36:02.074753 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.074729    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"]
Jul 09 18:36:02.074993 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.074967    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container" containerID="cri-o://f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374" gracePeriod=30
Jul 09 18:36:02.075096 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.075017    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kube-rbac-proxy" containerID="cri-o://8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae" gracePeriod=30
Jul 09 18:36:02.111651 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.111624    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"]
Jul 09 18:36:02.111969 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.111957    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kube-rbac-proxy"
Jul 09 18:36:02.112023 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.111970    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kube-rbac-proxy"
Jul 09 18:36:02.112023 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.111992    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container"
Jul 09 18:36:02.112023 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.111997    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container"
Jul 09 18:36:02.112132 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.112048    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kube-rbac-proxy"
Jul 09 18:36:02.112132 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.112057    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b014cfd-b839-4e4a-912f-81a85924b6bf" containerName="kserve-container"
Jul 09 18:36:02.115153 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.115133    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.118095 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.118073    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-05318-predictor-serving-cert\""
Jul 09 18:36:02.118209 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.118093    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-05318-kube-rbac-proxy-sar-config\""
Jul 09 18:36:02.128446 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.128427    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"]
Jul 09 18:36:02.141091 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.141071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d037bb5-cee9-4102-b7b1-764fc55d6198-proxy-tls\") pod \"error-404-isvc-05318-predictor-56fccfc459-lk7cr\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") " pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.141181 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.141116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdbj\" (UniqueName: \"kubernetes.io/projected/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-api-access-tvdbj\") pod \"error-404-isvc-05318-predictor-56fccfc459-lk7cr\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") " pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.141239 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.141215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-05318-predictor-56fccfc459-lk7cr\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") " pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.161245 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.161214    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:36:02.242383 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.242360    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-05318-predictor-56fccfc459-lk7cr\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") " pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.242500 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.242410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d037bb5-cee9-4102-b7b1-764fc55d6198-proxy-tls\") pod \"error-404-isvc-05318-predictor-56fccfc459-lk7cr\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") " pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.242500 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.242438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdbj\" (UniqueName: \"kubernetes.io/projected/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-api-access-tvdbj\") pod \"error-404-isvc-05318-predictor-56fccfc459-lk7cr\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") " pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.243003 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.242983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-05318-predictor-56fccfc459-lk7cr\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") " pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.244864 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.244833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d037bb5-cee9-4102-b7b1-764fc55d6198-proxy-tls\") pod \"error-404-isvc-05318-predictor-56fccfc459-lk7cr\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") " pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.260351 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.260332    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdbj\" (UniqueName: \"kubernetes.io/projected/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-api-access-tvdbj\") pod \"error-404-isvc-05318-predictor-56fccfc459-lk7cr\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") " pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.347394 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.347328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:36:02.425046 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.425023    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:02.462387 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.462356    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:36:02.593785 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.593754    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"]
Jul 09 18:36:02.596968 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:36:02.596945    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d037bb5_cee9_4102_b7b1_764fc55d6198.slice/crio-acf86080d0c96f698c486bb6f731d8ec3d1cc0a76c4b675698de218b2684d119 WatchSource:0}: Error finding container acf86080d0c96f698c486bb6f731d8ec3d1cc0a76c4b675698de218b2684d119: Status 404 returned error can't find the container with id acf86080d0c96f698c486bb6f731d8ec3d1cc0a76c4b675698de218b2684d119
Jul 09 18:36:02.705921 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.705893    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" event={"ID":"8d037bb5-cee9-4102-b7b1-764fc55d6198","Type":"ContainerStarted","Data":"da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c"}
Jul 09 18:36:02.706036 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.705948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" event={"ID":"8d037bb5-cee9-4102-b7b1-764fc55d6198","Type":"ContainerStarted","Data":"acf86080d0c96f698c486bb6f731d8ec3d1cc0a76c4b675698de218b2684d119"}
Jul 09 18:36:02.707519 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.707473    2560 generic.go:356] "Generic (PLEG): container finished" podID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerID="8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae" exitCode=2
Jul 09 18:36:02.707519 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:02.707508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" event={"ID":"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e","Type":"ContainerDied","Data":"8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae"}
Jul 09 18:36:03.714117 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:03.714087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" event={"ID":"8d037bb5-cee9-4102-b7b1-764fc55d6198","Type":"ContainerStarted","Data":"55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06"}
Jul 09 18:36:03.714570 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:03.714294    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:03.714570 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:03.714384    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:03.715704 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:03.715679    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 09 18:36:03.740050 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:03.740003    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" podStartSLOduration=1.739991174 podStartE2EDuration="1.739991174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:36:03.73896368 +0000 UTC m=+1907.890600965" watchObservedRunningTime="2026-07-09 18:36:03.739991174 +0000 UTC m=+1907.891628445"
Jul 09 18:36:04.718767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:04.718726    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 09 18:36:05.018722 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.018702    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:36:05.063627 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.063605    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-proxy-tls\") pod \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") "
Jul 09 18:36:05.063748 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.063646    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-rbac-proxy-sar-config\") pod \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") "
Jul 09 18:36:05.063748 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.063692    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw8zq\" (UniqueName: \"kubernetes.io/projected/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-api-access-lw8zq\") pod \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\" (UID: \"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e\") "
Jul 09 18:36:05.064096 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.064064    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" (UID: "66c6fbe6-a35e-4d92-b931-0f7c56aaa15e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:36:05.065606 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.065583    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" (UID: "66c6fbe6-a35e-4d92-b931-0f7c56aaa15e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:05.065682 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.065666    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-api-access-lw8zq" (OuterVolumeSpecName: "kube-api-access-lw8zq") pod "66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" (UID: "66c6fbe6-a35e-4d92-b931-0f7c56aaa15e"). InnerVolumeSpecName "kube-api-access-lw8zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:36:05.164405 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.164380    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8zq\" (UniqueName: \"kubernetes.io/projected/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-api-access-lw8zq\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:36:05.164405 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.164403    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:36:05.164556 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.164414    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:36:05.724469 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.724439    2560 generic.go:356] "Generic (PLEG): container finished" podID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerID="f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374" exitCode=0
Jul 09 18:36:05.724878 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.724508    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"
Jul 09 18:36:05.724878 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.724521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" event={"ID":"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e","Type":"ContainerDied","Data":"f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374"}
Jul 09 18:36:05.724878 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.724584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6" event={"ID":"66c6fbe6-a35e-4d92-b931-0f7c56aaa15e","Type":"ContainerDied","Data":"bc6e500f81922cdbdcac791cbc4bfecf28c5a66d8054afa821a3636eba0b9850"}
Jul 09 18:36:05.724878 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.724612    2560 scope.go:117] "RemoveContainer" containerID="8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae"
Jul 09 18:36:05.733567 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.733550    2560 scope.go:117] "RemoveContainer" containerID="f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374"
Jul 09 18:36:05.740587 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.740572    2560 scope.go:117] "RemoveContainer" containerID="8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae"
Jul 09 18:36:05.740832 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:36:05.740812    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae\": container with ID starting with 8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae not found: ID does not exist" containerID="8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae"
Jul 09 18:36:05.740905 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.740842    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae"} err="failed to get container status \"8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae\": rpc error: code = NotFound desc = could not find container \"8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae\": container with ID starting with 8b1f6221a5c59c2e921e4a05458c9b8f14c64865f825822c6e70fcd1f55c09ae not found: ID does not exist"
Jul 09 18:36:05.740905 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.740863    2560 scope.go:117] "RemoveContainer" containerID="f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374"
Jul 09 18:36:05.741122 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:36:05.741105    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374\": container with ID starting with f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374 not found: ID does not exist" containerID="f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374"
Jul 09 18:36:05.741163 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.741128    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374"} err="failed to get container status \"f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374\": rpc error: code = NotFound desc = could not find container \"f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374\": container with ID starting with f39c3f9d11615bed6fb825ede10f527b193e4dc126d9e332ab7550f1c7632374 not found: ID does not exist"
Jul 09 18:36:05.749991 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.749971    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"]
Jul 09 18:36:05.757394 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:05.757368    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05680-predictor-64bf589d5f-hrsh6"]
Jul 09 18:36:06.389266 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:06.389235    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" path="/var/lib/kubelet/pods/66c6fbe6-a35e-4d92-b931-0f7c56aaa15e/volumes"
Jul 09 18:36:09.541224 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:09.541190    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:36:09.722551 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:09.722522    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:36:09.722977 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:09.722951    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 09 18:36:19.722954 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:19.722918    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 09 18:36:21.611231 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:21.611184    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:36:21.860879 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:21.860840    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:36:29.723679 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:29.723632    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 09 18:36:39.723652 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:39.723611    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 09 18:36:49.723735 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:36:49.723697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:37:02.190779 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:37:02.190739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:37:02.332060 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:37:02.332009    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:37:47.302722 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:37:47.302688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:38:42.901074 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:38:42.900991    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:39:16.399537 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:39:16.399512    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:39:16.404795 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:39:16.404775    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:39:54.058356 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:39:54.058312    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:39:54.132881 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:39:54.132850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:44:16.425702 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:44:16.425672    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:44:16.431909 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:44:16.431888    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:44:30.762447 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:44:30.762408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:45:16.722635 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:16.722544    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:45:16.853958 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:16.853923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:45:16.868972 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:16.868942    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"]
Jul 09 18:45:16.869190 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:16.869166    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container" containerID="cri-o://da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c" gracePeriod=30
Jul 09 18:45:16.869244 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:16.869174    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kube-rbac-proxy" containerID="cri-o://55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06" gracePeriod=30
Jul 09 18:45:16.916612 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:16.916589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:45:17.079277 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:17.079252    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerID="55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06" exitCode=2
Jul 09 18:45:17.079375 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:17.079301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" event={"ID":"8d037bb5-cee9-4102-b7b1-764fc55d6198","Type":"ContainerDied","Data":"55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06"}
Jul 09 18:45:19.307923 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.307894    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:45:19.469804 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.469745    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvdbj\" (UniqueName: \"kubernetes.io/projected/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-api-access-tvdbj\") pod \"8d037bb5-cee9-4102-b7b1-764fc55d6198\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") "
Jul 09 18:45:19.469804 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.469800    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-rbac-proxy-sar-config\") pod \"8d037bb5-cee9-4102-b7b1-764fc55d6198\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") "
Jul 09 18:45:19.469995 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.469850    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d037bb5-cee9-4102-b7b1-764fc55d6198-proxy-tls\") pod \"8d037bb5-cee9-4102-b7b1-764fc55d6198\" (UID: \"8d037bb5-cee9-4102-b7b1-764fc55d6198\") "
Jul 09 18:45:19.470182 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.470148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d037bb5-cee9-4102-b7b1-764fc55d6198" (UID: "8d037bb5-cee9-4102-b7b1-764fc55d6198"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:45:19.471871 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.471844    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d037bb5-cee9-4102-b7b1-764fc55d6198-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d037bb5-cee9-4102-b7b1-764fc55d6198" (UID: "8d037bb5-cee9-4102-b7b1-764fc55d6198"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:45:19.471961 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.471876    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-api-access-tvdbj" (OuterVolumeSpecName: "kube-api-access-tvdbj") pod "8d037bb5-cee9-4102-b7b1-764fc55d6198" (UID: "8d037bb5-cee9-4102-b7b1-764fc55d6198"). InnerVolumeSpecName "kube-api-access-tvdbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:45:19.570690 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.570669    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d037bb5-cee9-4102-b7b1-764fc55d6198-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:45:19.570690 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.570688    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvdbj\" (UniqueName: \"kubernetes.io/projected/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-api-access-tvdbj\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:45:19.570816 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:19.570697    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d037bb5-cee9-4102-b7b1-764fc55d6198-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:45:20.093180 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.093151    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerID="da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c" exitCode=0
Jul 09 18:45:20.093290 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.093196    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"
Jul 09 18:45:20.093290 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.093214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" event={"ID":"8d037bb5-cee9-4102-b7b1-764fc55d6198","Type":"ContainerDied","Data":"da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c"}
Jul 09 18:45:20.093290 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.093266    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr" event={"ID":"8d037bb5-cee9-4102-b7b1-764fc55d6198","Type":"ContainerDied","Data":"acf86080d0c96f698c486bb6f731d8ec3d1cc0a76c4b675698de218b2684d119"}
Jul 09 18:45:20.093290 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.093291    2560 scope.go:117] "RemoveContainer" containerID="55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06"
Jul 09 18:45:20.101417 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.101402    2560 scope.go:117] "RemoveContainer" containerID="da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c"
Jul 09 18:45:20.108305 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.108291    2560 scope.go:117] "RemoveContainer" containerID="55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06"
Jul 09 18:45:20.108556 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:45:20.108538    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06\": container with ID starting with 55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06 not found: ID does not exist" containerID="55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06"
Jul 09 18:45:20.108613 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.108563    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06"} err="failed to get container status \"55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06\": rpc error: code = NotFound desc = could not find container \"55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06\": container with ID starting with 55eed081feb209b1f02ddeaf11f5e7f805765be4e8b942cb37d0f29848645d06 not found: ID does not exist"
Jul 09 18:45:20.108613 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.108580    2560 scope.go:117] "RemoveContainer" containerID="da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c"
Jul 09 18:45:20.108793 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:45:20.108780    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c\": container with ID starting with da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c not found: ID does not exist" containerID="da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c"
Jul 09 18:45:20.108833 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.108795    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c"} err="failed to get container status \"da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c\": rpc error: code = NotFound desc = could not find container \"da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c\": container with ID starting with da6994b6475a4ae212849a58536f6394cae61e784a40045203e6fd5d31a3462c not found: ID does not exist"
Jul 09 18:45:20.115259 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.115240    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"]
Jul 09 18:45:20.121110 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.121090    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-05318-predictor-56fccfc459-lk7cr"]
Jul 09 18:45:20.388743 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:45:20.388685    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" path="/var/lib/kubelet/pods/8d037bb5-cee9-4102-b7b1-764fc55d6198/volumes"
Jul 09 18:47:17.877104 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:47:17.877069    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:49:16.455817 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:49:16.455726    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:49:16.464327 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:49:16.464308    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:51:50.629040 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:51:50.628997    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:52:40.590774 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:40.590743    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:52:40.743762 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:40.743731    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:52:40.773462 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:40.773434    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"]
Jul 09 18:52:40.773750 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:40.773709    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container" containerID="cri-o://38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a" gracePeriod=30
Jul 09 18:52:40.773901 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:40.773769    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kube-rbac-proxy" containerID="cri-o://60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5" gracePeriod=30
Jul 09 18:52:40.819418 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:40.819385    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:52:41.872632 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:41.872593    2560 generic.go:356] "Generic (PLEG): container finished" podID="ab903443-f510-4930-ab86-f308ef331e65" containerID="60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5" exitCode=2
Jul 09 18:52:41.873033 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:41.872669    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" event={"ID":"ab903443-f510-4930-ab86-f308ef331e65","Type":"ContainerDied","Data":"60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5"}
Jul 09 18:52:42.144528 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:42.144431    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:52:42.240053 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:42.240020    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:52:43.127511 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.127467    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:52:43.264259 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.264237    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab903443-f510-4930-ab86-f308ef331e65-proxy-tls\") pod \"ab903443-f510-4930-ab86-f308ef331e65\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") "
Jul 09 18:52:43.264367 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.264281    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab903443-f510-4930-ab86-f308ef331e65-kube-rbac-proxy-sar-config\") pod \"ab903443-f510-4930-ab86-f308ef331e65\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") "
Jul 09 18:52:43.264367 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.264311    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crvxf\" (UniqueName: \"kubernetes.io/projected/ab903443-f510-4930-ab86-f308ef331e65-kube-api-access-crvxf\") pod \"ab903443-f510-4930-ab86-f308ef331e65\" (UID: \"ab903443-f510-4930-ab86-f308ef331e65\") "
Jul 09 18:52:43.264626 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.264605    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab903443-f510-4930-ab86-f308ef331e65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab903443-f510-4930-ab86-f308ef331e65" (UID: "ab903443-f510-4930-ab86-f308ef331e65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:52:43.266315 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.266294    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab903443-f510-4930-ab86-f308ef331e65-kube-api-access-crvxf" (OuterVolumeSpecName: "kube-api-access-crvxf") pod "ab903443-f510-4930-ab86-f308ef331e65" (UID: "ab903443-f510-4930-ab86-f308ef331e65"). InnerVolumeSpecName "kube-api-access-crvxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:52:43.266421 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.266404    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab903443-f510-4930-ab86-f308ef331e65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab903443-f510-4930-ab86-f308ef331e65" (UID: "ab903443-f510-4930-ab86-f308ef331e65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:52:43.365718 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.365695    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab903443-f510-4930-ab86-f308ef331e65-proxy-tls\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:52:43.365718 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.365716    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab903443-f510-4930-ab86-f308ef331e65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:52:43.365842 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.365726    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvxf\" (UniqueName: \"kubernetes.io/projected/ab903443-f510-4930-ab86-f308ef331e65-kube-api-access-crvxf\") on node \"ip-10-0-135-87.ec2.internal\" DevicePath \"\""
Jul 09 18:52:43.885964 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.885932    2560 generic.go:356] "Generic (PLEG): container finished" podID="ab903443-f510-4930-ab86-f308ef331e65" containerID="38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a" exitCode=0
Jul 09 18:52:43.886126 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.885992    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"
Jul 09 18:52:43.886126 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.886021    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" event={"ID":"ab903443-f510-4930-ab86-f308ef331e65","Type":"ContainerDied","Data":"38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a"}
Jul 09 18:52:43.886126 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.886089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt" event={"ID":"ab903443-f510-4930-ab86-f308ef331e65","Type":"ContainerDied","Data":"a2a3dbc97960db2a50dff7c2952fa2ac6d932c6fba73a733402ad9809a333625"}
Jul 09 18:52:43.886126 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.886121    2560 scope.go:117] "RemoveContainer" containerID="60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5"
Jul 09 18:52:43.895644 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.895624    2560 scope.go:117] "RemoveContainer" containerID="38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a"
Jul 09 18:52:43.903513 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.903492    2560 scope.go:117] "RemoveContainer" containerID="60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5"
Jul 09 18:52:43.903826 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:52:43.903808    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5\": container with ID starting with 60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5 not found: ID does not exist" containerID="60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5"
Jul 09 18:52:43.903878 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.903835    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5"} err="failed to get container status \"60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5\": rpc error: code = NotFound desc = could not find container \"60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5\": container with ID starting with 60fb6f3f0bdc398747c34a715cf69f70e29898c775d9b842f4d081677ecebfd5 not found: ID does not exist"
Jul 09 18:52:43.903878 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.903852    2560 scope.go:117] "RemoveContainer" containerID="38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a"
Jul 09 18:52:43.904118 ip-10-0-135-87 kubenswrapper[2560]: E0709 18:52:43.904099    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a\": container with ID starting with 38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a not found: ID does not exist" containerID="38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a"
Jul 09 18:52:43.904173 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.904129    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a"} err="failed to get container status \"38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a\": rpc error: code = NotFound desc = could not find container \"38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a\": container with ID starting with 38fac807e9faa9d6525d791d9fd03db4d31153c00d19f7e3a5676224a9dd7d2a not found: ID does not exist"
Jul 09 18:52:43.907706 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.907681    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"]
Jul 09 18:52:43.911435 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:43.911411    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-95b80-predictor-6d45f8d7f7-hlhtt"]
Jul 09 18:52:44.388720 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:52:44.388690    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab903443-f510-4930-ab86-f308ef331e65" path="/var/lib/kubelet/pods/ab903443-f510-4930-ab86-f308ef331e65/volumes"
Jul 09 18:53:12.043524 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:12.043489    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:12.222064 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:12.222039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:12.321217 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:12.321148    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:14.812281 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:14.812246    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5wc7j_c19aec6c-4cdd-41dc-b6b5-3b5adfe023a8/global-pull-secret-syncer/0.log"
Jul 09 18:53:14.962854 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:14.962825    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log"
Jul 09 18:53:15.077056 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:15.077000    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-87.ec2.internal_6f2dcd2c4647c8a5c8152fee259ad6a9/haproxy/0.log"
Jul 09 18:53:18.514845 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:18.514804    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:18.615449 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:18.615420    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8hb8n_b98744d7-a4b6-4f0e-830b-da6c2ba8bbb9/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:19.341579 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.341551    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e6c9aafc-8bcb-4ee8-a984-663d076dba1e/alertmanager/0.log"
Jul 09 18:53:19.381612 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.381587    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e6c9aafc-8bcb-4ee8-a984-663d076dba1e/config-reloader/0.log"
Jul 09 18:53:19.411652 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.411624    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e6c9aafc-8bcb-4ee8-a984-663d076dba1e/kube-rbac-proxy-web/0.log"
Jul 09 18:53:19.444100 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.444077    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e6c9aafc-8bcb-4ee8-a984-663d076dba1e/kube-rbac-proxy/0.log"
Jul 09 18:53:19.471753 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.471732    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e6c9aafc-8bcb-4ee8-a984-663d076dba1e/kube-rbac-proxy-metric/0.log"
Jul 09 18:53:19.513743 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.513721    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e6c9aafc-8bcb-4ee8-a984-663d076dba1e/prom-label-proxy/0.log"
Jul 09 18:53:19.539063 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.539045    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e6c9aafc-8bcb-4ee8-a984-663d076dba1e/init-config-reloader/0.log"
Jul 09 18:53:19.717435 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.717378    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d5c4444c8-4qwcq_4ec1d066-e324-4326-bf9b-d002fcdb3f80/metrics-server/0.log"
Jul 09 18:53:19.834429 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.834404    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9f4bn_1d3414c2-9767-4569-bb59-23f3f52d13a9/node-exporter/0.log"
Jul 09 18:53:19.873130 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.873105    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9f4bn_1d3414c2-9767-4569-bb59-23f3f52d13a9/kube-rbac-proxy/0.log"
Jul 09 18:53:19.898738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:19.898717    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9f4bn_1d3414c2-9767-4569-bb59-23f3f52d13a9/init-textfile/0.log"
Jul 09 18:53:20.652637 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:20.652608    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cdc4c-998tg_9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5/thanos-query/0.log"
Jul 09 18:53:20.681912 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:20.681884    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cdc4c-998tg_9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5/kube-rbac-proxy-web/0.log"
Jul 09 18:53:20.709360 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:20.709343    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cdc4c-998tg_9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5/kube-rbac-proxy/0.log"
Jul 09 18:53:20.734561 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:20.734532    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cdc4c-998tg_9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5/prom-label-proxy/0.log"
Jul 09 18:53:20.774318 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:20.774296    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cdc4c-998tg_9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5/kube-rbac-proxy-rules/0.log"
Jul 09 18:53:20.808717 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:20.808697    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f87cdc4c-998tg_9f6b00d6-56a4-4e47-bb07-8fae0fafa5d5/kube-rbac-proxy-metrics/0.log"
Jul 09 18:53:23.445684 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.445649    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"]
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.445971    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kube-rbac-proxy"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.445983    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kube-rbac-proxy"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.445993    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.445999    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446015    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kube-rbac-proxy"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446021    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kube-rbac-proxy"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446029    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kube-rbac-proxy"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446034    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kube-rbac-proxy"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446043    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446048    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446059    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container"
Jul 09 18:53:23.446086 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446064    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container"
Jul 09 18:53:23.446515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446110    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kube-rbac-proxy"
Jul 09 18:53:23.446515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446118    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kube-rbac-proxy"
Jul 09 18:53:23.446515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446125    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kube-rbac-proxy"
Jul 09 18:53:23.446515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446144    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab903443-f510-4930-ab86-f308ef331e65" containerName="kserve-container"
Jul 09 18:53:23.446515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446151    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c6fbe6-a35e-4d92-b931-0f7c56aaa15e" containerName="kserve-container"
Jul 09 18:53:23.446515 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.446158    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d037bb5-cee9-4102-b7b1-764fc55d6198" containerName="kserve-container"
Jul 09 18:53:23.450379 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.450356    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.453218 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.453196    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gkb77\"/\"kube-root-ca.crt\""
Jul 09 18:53:23.454045 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.454024    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gkb77\"/\"default-dockercfg-9z2x5\""
Jul 09 18:53:23.454230 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.454027    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gkb77\"/\"openshift-service-ca.crt\""
Jul 09 18:53:23.461850 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.461827    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"]
Jul 09 18:53:23.544846 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.544823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6b6h\" (UniqueName: \"kubernetes.io/projected/c5b428e7-62f8-4f26-9c52-0201df86ae2d-kube-api-access-d6b6h\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.544971 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.544853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-proc\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.544971 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.544884    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-podres\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.545062 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.544987    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-sys\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.545062 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.545041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-lib-modules\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.645271 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.645245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-podres\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.645389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.645285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-sys\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.645389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.645319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-lib-modules\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.645389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.645353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6b6h\" (UniqueName: \"kubernetes.io/projected/c5b428e7-62f8-4f26-9c52-0201df86ae2d-kube-api-access-d6b6h\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.645389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.645371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-proc\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.645389 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.645384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-sys\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.645694 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.645449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-proc\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.645694 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.645511    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-podres\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.645694 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.645524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5b428e7-62f8-4f26-9c52-0201df86ae2d-lib-modules\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.653686 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.653664    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6b6h\" (UniqueName: \"kubernetes.io/projected/c5b428e7-62f8-4f26-9c52-0201df86ae2d-kube-api-access-d6b6h\") pod \"perf-node-gather-daemonset-55jzf\" (UID: \"c5b428e7-62f8-4f26-9c52-0201df86ae2d\") " pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.761111 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.761089    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:23.892767 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:23.892744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"]
Jul 09 18:53:23.894986 ip-10-0-135-87 kubenswrapper[2560]: W0709 18:53:23.894955    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc5b428e7_62f8_4f26_9c52_0201df86ae2d.slice/crio-22a76e92788053073410f2e3fc645e28a1245f87717968e447585e2eea76226a WatchSource:0}: Error finding container 22a76e92788053073410f2e3fc645e28a1245f87717968e447585e2eea76226a: Status 404 returned error can't find the container with id 22a76e92788053073410f2e3fc645e28a1245f87717968e447585e2eea76226a
Jul 09 18:53:24.069887 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:24.069837    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf" event={"ID":"c5b428e7-62f8-4f26-9c52-0201df86ae2d","Type":"ContainerStarted","Data":"deedb208bdb65d9c07bdd872c2310edc55fc46817fbd870a5dc9de32c3db1feb"}
Jul 09 18:53:24.069887 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:24.069881    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf" event={"ID":"c5b428e7-62f8-4f26-9c52-0201df86ae2d","Type":"ContainerStarted","Data":"22a76e92788053073410f2e3fc645e28a1245f87717968e447585e2eea76226a"}
Jul 09 18:53:24.070029 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:24.069926    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:24.091293 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:24.091254    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf" podStartSLOduration=1.091241074 podStartE2EDuration="1.091241074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:24.089910153 +0000 UTC m=+2948.241547443" watchObservedRunningTime="2026-07-09 18:53:24.091241074 +0000 UTC m=+2948.242878344"
Jul 09 18:53:24.620994 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:24.620965    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zrxtr_d40f86e5-325b-4c1f-8468-59b44d58b0c4/dns/0.log"
Jul 09 18:53:24.646195 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:24.646125    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zrxtr_d40f86e5-325b-4c1f-8468-59b44d58b0c4/kube-rbac-proxy/0.log"
Jul 09 18:53:24.680805 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:24.680784    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5xkmx_9afa7d8f-5b83-494b-9166-c8fd3fb1282b/dns-node-resolver/0.log"
Jul 09 18:53:25.229324 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:25.229301    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b74d8bc8-dg69n_1e0cc46f-0a9d-4139-adb0-969e5056fe79/registry/0.log"
Jul 09 18:53:25.278559 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:25.278532    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f6mv8_c33e40f4-ff09-4aa9-9fb1-7c8de50e683a/node-ca/0.log"
Jul 09 18:53:26.577738 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:26.577706    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mhsft_0f45cd8f-7df7-4798-a79e-b292782eef6f/serve-healthcheck-canary/0.log"
Jul 09 18:53:26.996455 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:26.996404    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jnp7h_4e494343-ee4c-4a4f-8989-24a3e4bda412/kube-rbac-proxy/0.log"
Jul 09 18:53:27.033431 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:27.033409    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jnp7h_4e494343-ee4c-4a4f-8989-24a3e4bda412/exporter/0.log"
Jul 09 18:53:27.074053 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:27.074029    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jnp7h_4e494343-ee4c-4a4f-8989-24a3e4bda412/extractor/0.log"
Jul 09 18:53:29.689719 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:29.689689    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-57b57d8c6-22rcv_964d20ad-1d5b-43e8-9407-3248762445c5/manager/0.log"
Jul 09 18:53:29.722939 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:29.722916    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-xb5ts_01817576-2fb0-4596-a7ad-a940d5c458b3/manager/0.log"
Jul 09 18:53:29.754236 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:29.754215    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-sw9vg_9f3533f2-0ecf-442f-9ea4-9c1fc338a031/server/0.log"
Jul 09 18:53:29.848663 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:29.848635    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-trcc8_a92932da-b8c0-4eb9-926b-b960ddf5feb7/manager/0.log"
Jul 09 18:53:30.084103 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:30.084076    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gkb77/perf-node-gather-daemonset-55jzf"
Jul 09 18:53:34.235616 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:34.235538    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-tjczp_14651ff7-a827-44f9-9e24-612d360786cf/migrator/0.log"
Jul 09 18:53:34.262394 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:34.262374    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-tjczp_14651ff7-a827-44f9-9e24-612d360786cf/graceful-termination/0.log"
Jul 09 18:53:35.687312 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:35.687286    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7gzr4_bb158c6c-ad86-4462-a1e0-fe975d0055c4/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:53:35.710768 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:35.710745    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7gzr4_bb158c6c-ad86-4462-a1e0-fe975d0055c4/egress-router-binary-copy/0.log"
Jul 09 18:53:35.734957 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:35.734884    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7gzr4_bb158c6c-ad86-4462-a1e0-fe975d0055c4/cni-plugins/0.log"
Jul 09 18:53:35.759922 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:35.759901    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7gzr4_bb158c6c-ad86-4462-a1e0-fe975d0055c4/bond-cni-plugin/0.log"
Jul 09 18:53:35.784736 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:35.784718    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7gzr4_bb158c6c-ad86-4462-a1e0-fe975d0055c4/routeoverride-cni/0.log"
Jul 09 18:53:35.810017 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:35.809993    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7gzr4_bb158c6c-ad86-4462-a1e0-fe975d0055c4/whereabouts-cni-bincopy/0.log"
Jul 09 18:53:35.838199 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:35.838159    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7gzr4_bb158c6c-ad86-4462-a1e0-fe975d0055c4/whereabouts-cni/0.log"
Jul 09 18:53:36.225923 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:36.225900    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mm9zg_8220436b-a8e2-4f0e-ab53-776cc9660d6a/kube-multus/0.log"
Jul 09 18:53:36.368056 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:36.368036    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zsbvk_1803cc34-6cae-4580-a034-b67c1cab8b8e/network-metrics-daemon/0.log"
Jul 09 18:53:36.389972 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:36.389952    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zsbvk_1803cc34-6cae-4580-a034-b67c1cab8b8e/kube-rbac-proxy/0.log"
Jul 09 18:53:37.380281 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:37.380255    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-controller/0.log"
Jul 09 18:53:37.402025 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:37.402003    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/1.log"
Jul 09 18:53:37.402277 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:37.402258    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovn-acl-logging/0.log"
Jul 09 18:53:37.424607 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:37.424586    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/kube-rbac-proxy-node/0.log"
Jul 09 18:53:37.449754 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:37.449729    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:53:37.472789 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:37.472767    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/northd/0.log"
Jul 09 18:53:37.498760 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:37.498744    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/nbdb/0.log"
Jul 09 18:53:37.522904 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:37.522884    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/sbdb/0.log"
Jul 09 18:53:37.571253 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:37.571230    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7npv_f4918fd2-0cae-42eb-b8dc-855565f0298b/ovnkube-controller/0.log"
Jul 09 18:53:39.023133 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:39.023065    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-gbg4t_882b7841-7d01-46cb-afca-f21a77389624/network-check-target-container/0.log"
Jul 09 18:53:40.007899 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:40.007878    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-75tj7_0265afbf-1a11-438e-b093-11f6dcfbd075/iptables-alerter/0.log"
Jul 09 18:53:40.770371 ip-10-0-135-87 kubenswrapper[2560]: I0709 18:53:40.770344    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rvc7q_1b4516b8-9d8d-48ab-b802-794ad1ed6874/tuned/0.log"