Jul 28 12:53:48.668599 ip-10-0-135-149 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:53:49.012603 ip-10-0-135-149 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:49.012603 ip-10-0-135-149 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:53:49.012603 ip-10-0-135-149 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:49.012603 ip-10-0-135-149 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:53:49.012603 ip-10-0-135-149 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:49.014369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.014239    2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:53:49.020422 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020406    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:49.020422 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020422    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020426    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020429    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020432    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020435    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020438    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020440    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020445    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020449    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020452    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020454    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020463    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020466    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020469    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020471    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020474    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020476    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020480    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020484    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:49.020484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020487    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020490    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020493    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020496    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020499    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020501    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020504    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020507    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020509    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020511    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020514    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020516    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020519    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020521    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020524    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020526    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020529    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020531    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020534    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:49.020923 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020537    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020539    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020542    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020544    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020547    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020549    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020551    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020554    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020557    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020559    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020561    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020564    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020567    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020571    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020574    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020577    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020580    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020582    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020585    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:49.021416 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020587    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020590    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020592    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020595    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020597    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020600    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020602    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020604    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020607    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020609    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020612    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020614    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020617    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020620    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020622    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020625    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020629    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020631    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020634    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020636    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:49.021869 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020639    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020642    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020644    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020647    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020649    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020651    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020655    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020657    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020660    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020663    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020665    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020668    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020670    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020673    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020675    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020678    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020680    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020683    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020685    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020688    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:49.022336 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.020690    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022624    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022632    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022635    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022638    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022641    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022644    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022647    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022650    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022652    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022654    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022657    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022660    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022663    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022665    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022668    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022670    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022672    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022675    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022677    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:49.022834 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022680    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022682    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022685    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022687    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022690    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022692    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022695    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022697    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022700    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022702    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022704    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022707    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022709    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022712    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022716    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022718    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022721    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022723    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022726    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022729    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:49.023317 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022731    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022734    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022736    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022738    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022741    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022745    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022748    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022751    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022754    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022757    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022759    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022761    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022764    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022766    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022769    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022771    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022774    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022776    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022779    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:49.023841 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022782    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022784    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022787    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022789    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022792    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022794    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022797    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022800    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022802    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022805    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022808    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022810    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022812    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022815    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022818    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022820    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022823    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022825    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022829    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:49.024283 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022832    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022835    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022838    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022840    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022842    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022845    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022848    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022850    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022852    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022855    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022857    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022859    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022864    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022867    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022870    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022873    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022875    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022878    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022881    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:49.024741 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022884    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022887    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.022890    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023237    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023242    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023245    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023247    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023250    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023253    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023255    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023258    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023261    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023263    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023266    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023268    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023270    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023273    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023275    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023278    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023280    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:49.025181 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023283    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023285    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023287    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023290    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023292    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023295    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023298    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023300    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023303    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023306    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023308    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023311    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023313    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023316    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023319    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023321    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023324    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023327    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023329    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023332    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:49.025653 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023335    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023338    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023340    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023358    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023361    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023363    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023365    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023368    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023371    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023373    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023376    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023378    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023381    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023383    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023386    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023388    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023391    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023394    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023396    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023398    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:49.026116 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023401    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023404    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023406    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023408    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023410    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023413    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023417    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023419    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023422    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023424    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023427    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023429    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023431    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023434    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023437    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023439    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023441    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023444    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023446    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:49.026592 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023448    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023451    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023455    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023458    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023461    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023464    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023466    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023469    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023471    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023476    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023479    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023483    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023486    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023489    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023492    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023494    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023497    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023500    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023503    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:49.027055 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023506    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023509    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023512    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.023514    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023588    2553 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023602    2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023614    2553 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023619    2553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023624    2553 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023627    2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023631    2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023635    2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023638    2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023641    2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023644    2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023648    2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023651    2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023653    2553 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023656    2553 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023659    2553 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023662    2553 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023665    2553 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023671    2553 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023674    2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:53:49.027502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023677    2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023680    2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023684    2553 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023687    2553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023690    2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023693    2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023696    2553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023699    2553 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023702    2553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023705    2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023708    2553 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023711    2553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023714    2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023717    2553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023720    2553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023722    2553 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023725    2553 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023736    2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023742    2553 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023745    2553 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023747    2553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023750    2553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023753    2553 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023757    2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:53:49.028081 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023759    2553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023762    2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023765    2553 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023768    2553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023771    2553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023774    2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023776    2553 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023779    2553 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023782    2553 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023785    2553 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023789    2553 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023791    2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023794    2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023797    2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023800    2553 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023803    2553 flags.go:64] FLAG: --help="false"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023806    2553 flags.go:64] FLAG: --hostname-override="ip-10-0-135-149.ec2.internal"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023809    2553 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023812    2553 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023815    2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023818    2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023821    2553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023824    2553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:53:49.028640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023826    2553 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023829    2553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023832    2553 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023835    2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023838    2553 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023841    2553 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023844    2553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023849    2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023853    2553 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023856    2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023859    2553 flags.go:64] FLAG: --lock-file=""
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023861    2553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023864    2553 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023867    2553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023872    2553 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023874    2553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023877    2553 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023880    2553 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023883    2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023886    2553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023889    2553 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023891    2553 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023899    2553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023902    2553 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023906    2553 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:53:49.029214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023909    2553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023912    2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023915    2553 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023918    2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023921    2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023924    2553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023926    2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023933    2553 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023936    2553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023939    2553 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023942    2553 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023945    2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023950    2553 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023953    2553 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023957    2553 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023960    2553 flags.go:64] FLAG: --port="10250"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023963    2553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023966    2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b6bc3d4cc92f6c61"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023969    2553 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023972    2553 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023975    2553 flags.go:64] FLAG: --register-node="true"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023978    2553 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023981    2553 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023985    2553 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:53:49.029833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023988    2553 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023990    2553 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023994    2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.023997    2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024000    2553 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024003    2553 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024007    2553 flags.go:64] FLAG: --runonce="false"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024010    2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024012    2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024015    2553 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024018    2553 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024021    2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024023    2553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024026    2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024029    2553 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024032    2553 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024034    2553 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024037    2553 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024040    2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024042    2553 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024046    2553 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024049    2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024058    2553 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024061    2553 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024064    2553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:53:49.030456 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024067    2553 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024070    2553 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024073    2553 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024075    2553 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024078    2553 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024095    2553 flags.go:64] FLAG: --v="2"
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024100    2553 flags.go:64] FLAG: --version="false"
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024105    2553 flags.go:64] FLAG: --vmodule=""
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024108    2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.024112    2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024222    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024226    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024229    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024233    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024236    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024239    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024241    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024244    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024247    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024249    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024252    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024255    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:49.031036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024257    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024260    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024262    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024265    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024267    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024270    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024273    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024276    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024279    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024281    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024284    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024288    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024291    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024294    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024297    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024300    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024302    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024305    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024308    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024310    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:49.031557 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024312    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024315    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024317    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024322    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024324    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024327    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024330    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024357    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024364    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024369    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024372    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024375    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024378    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024381    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024383    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024386    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024389    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024392    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024394    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024399    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:49.032036 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024403    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024407    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024410    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024413    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024415    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024418    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024421    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024423    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024426    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024428    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024431    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024433    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024436    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024438    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024441    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024445    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024448    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024450    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024453    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024455    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:49.032555 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024458    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024461    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024463    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024466    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024468    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024471    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024473    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024476    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024479    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024481    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024483    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024487    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024491    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024493    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024497    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024500    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024503    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024506    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024509    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:49.033025 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024511    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024513    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024516    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024518    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024521    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024523    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024526    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.024528    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.025002    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.031423    2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.031438    2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.032074    2553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.032082    2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032135    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032141    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:49.033484 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032144    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032147    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032150    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032153    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032156    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032159    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032162    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032164    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032166    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032169    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032171    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032174    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032177    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032181    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032184    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032187    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032190    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032192    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032195    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:49.033856 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032197    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032200    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032202    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032205    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032209    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032212    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032215    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032218    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032223    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032225    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032228    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032231    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032233    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032236    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032239    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032241    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032244    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032247    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032249    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032252    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:49.034304 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032254    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032257    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032260    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032262    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032265    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032267    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032270    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032272    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032275    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032277    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032281    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032284    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032287    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032290    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032292    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032294    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032297    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032299    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032302    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:49.034844 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032305    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032308    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032311    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032313    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032316    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032318    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032321    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032323    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032325    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032328    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032330    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032333    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032336    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032338    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032340    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032353    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032356    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032358    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032360    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032363    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:49.035281 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032366    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032368    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032370    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032373    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032375    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032378    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032380    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032383    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032386    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032388    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032391    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032393    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032396    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032399    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032401    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032404    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032407    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032410    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:49.035828 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032412    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.032417    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032523    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032528    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032531    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032534    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032537    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032539    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032542    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032545    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032547    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032550    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032553    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032555    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032558    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:49.036248 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032560    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032563    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032565    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032569    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032573    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032576    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032579    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032582    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032584    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032587    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032590    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032593    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032596    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032599    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032602    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032605    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032607    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032610    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032612    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032615    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:49.036622 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032617    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032619    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032622    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032624    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032627    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032629    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032631    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032634    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032636    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032639    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032641    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032644    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032646    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032649    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032651    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032654    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032656    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032659    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032661    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:49.037095 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032664    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032666    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032668    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032671    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032674    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032676    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032679    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032681    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032684    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032692    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032694    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032697    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032699    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032701    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032704    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032706    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032709    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032711    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032714    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:49.037537 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032716    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032719    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032722    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032725    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032728    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032731    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032733    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032736    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032738    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032741    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032743    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032746    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032748    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032750    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032753    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032755    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032758    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032761    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032763    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:49.037984 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032766    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032768    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032771    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032773    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032776    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032778    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032781    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032783    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:49.032787    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.032791    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.032890    2553 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.034756    2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.035938    2553 server.go:1019] "Starting client certificate rotation"
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.036039    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:53:49.038543 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.036078    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:53:49.038877 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.038129    2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:53:49.042996 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.042975    2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:53:49.049281 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.049264    2553 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:53:49.052537 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.052521    2553 log.go:25] "Validated CRI v1 image API"
Jul 28 12:53:49.053303 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.053283    2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:53:49.056268 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.056246    2553 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c0ec6715-b6f8-46b6-a030-7feb2a15a762:/dev/nvme0n1p4 e3ba54df-e5d6-4722-b36b-65ac92a84774:/dev/nvme0n1p3]
Jul 28 12:53:49.056362 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.056266    2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:53:49.061146 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.061038    2553 manager.go:216] Machine: {Timestamp:2026-07-28 12:53:49.059848415 +0000 UTC m=+0.290666072 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200187 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2705bc14c927f84476b24e5217b5d6 SystemUUID:ec2705bc-14c9-27f8-4476-b24e5217b5d6 BootID:ffcca9cf-d453-4443-8f56-4be7366f168e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cf:1b:57:3c:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cf:1b:57:3c:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:aa:2c:60:0d:39 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:53:49.061146 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.061138    2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:53:49.061296 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.061274    2553 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:53:49.064020 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.063993    2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:53:49.064178 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.064024    2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:53:49.065322 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.065311    2553 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:53:49.065403 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.065328    2553 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:53:49.065403 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.065340    2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:53:49.066042 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.066031    2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:53:49.066816 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.066805    2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:53:49.068409 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.068396    2553 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:53:49.068535 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.068525    2553 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:53:49.068598 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.068561    2553 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:53:49.068598 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.068575    2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:53:49.068598 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.068596    2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:53:49.068714 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.068609    2553 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:53:49.068714 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.068620    2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:53:49.068988 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.068975    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:53:49.069053 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.068998    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:53:49.070374 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.070324    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:53:49.070989 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.070974    2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:53:49.072763 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.072746    2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:53:49.072840 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.072787    2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:53:49.074265 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074250    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:53:49.074332 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074270    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:53:49.074332 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074280    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:53:49.074332 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074289    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:53:49.074332 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074303    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:53:49.074332 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074311    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:53:49.074332 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074324    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:53:49.074332 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074332    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:53:49.074579 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074357    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:53:49.074579 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074366    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:53:49.074579 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074379    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:53:49.074579 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.074394    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:53:49.075131 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.075120    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:53:49.075179 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.075133    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:53:49.078171 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.078155    2553 server.go:1268] "Started kubelet"
Jul 28 12:53:49.082046 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.081979    2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:53:49.082141 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.082095    2553 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:53:49.082376 ip-10-0-135-149 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:53:49.082706 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.082650    2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:53:49.084375 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.084359    2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:53:49.084758 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.084608    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:53:49.085712 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.085463    2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-149.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:53:49.085712 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.085483    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:53:49.086100 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.086085    2553 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:53:49.088431 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.088401    2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:53:49.088808 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.088026    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-149.ec2.internal.18c675421adf74c1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-149.ec2.internal,UID:ip-10-0-135-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-149.ec2.internal,},FirstTimestamp:2026-07-28 12:53:49.078123713 +0000 UTC m=+0.308941370,LastTimestamp:2026-07-28 12:53:49.078123713 +0000 UTC m=+0.308941370,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-149.ec2.internal,}"
Jul 28 12:53:49.090971 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.090956    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-927nw"
Jul 28 12:53:49.091533 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.091519    2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:53:49.091627 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.091614    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:53:49.091679 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.091650    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:49.092187 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092171    2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:53:49.092268 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092222    2553 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:53:49.092268 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092234    2553 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:53:49.092374 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092320    2553 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:53:49.092374 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092331    2553 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:53:49.092699 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092680    2553 factory.go:153] Registering CRI-O factory
Jul 28 12:53:49.092699 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092699    2553 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:53:49.092829 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092741    2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:53:49.092829 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092750    2553 factory.go:55] Registering systemd factory
Jul 28 12:53:49.092829 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092758    2553 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:53:49.092829 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092780    2553 factory.go:103] Registering Raw factory
Jul 28 12:53:49.092829 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.092789    2553 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:53:49.093168 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.093156    2553 manager.go:349] Starting recovery of all containers
Jul 28 12:53:49.096079 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.096056    2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 12:53:49.096079 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.096069    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 12:53:49.101027 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.101006    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-927nw"
Jul 28 12:53:49.103736 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.103713    2553 manager.go:354] Recovery completed
Jul 28 12:53:49.105200 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.105183    2553 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 12:53:49.108821 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.108800    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:49.110585 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.110566    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:49.110656 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.110602    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:49.110656 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.110617    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:49.111092 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.111076    2553 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:53:49.111092 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.111087    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:49.111212 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.111089    2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:53:49.111212 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.111127    2553 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:53:49.111801 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.111786    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:49.111856 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.111813    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:49.111856 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.111827    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:49.112797 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.112786    2553 policy_none.go:49] "None policy: Start"
Jul 28 12:53:49.112837 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.112800    2553 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:53:49.112837 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.112811    2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:53:49.113081 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.113011    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-149.ec2.internal.18c675421ccec75e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-149.ec2.internal,UID:ip-10-0-135-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-149.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-149.ec2.internal,},FirstTimestamp:2026-07-28 12:53:49.110585182 +0000 UTC m=+0.341402846,LastTimestamp:2026-07-28 12:53:49.110585182 +0000 UTC m=+0.341402846,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-149.ec2.internal,}"
Jul 28 12:53:49.113718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.113707    2553 policy_none.go:47] "Start"
Jul 28 12:53:49.152038 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.151896    2553 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:53:49.152161 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.152073    2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:53:49.152161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.152088    2553 server.go:87] "Starting device plugin registration server"
Jul 28 12:53:49.152380 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.152367    2553 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:53:49.152453 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.152384    2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:53:49.152453 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.152449    2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:53:49.152543 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.152533    2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:53:49.152587 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.152545    2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:53:49.153068 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.153047    2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:53:49.153130 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.153104    2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:49.252497 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.252473    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:49.254214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.253670    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:49.254214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.253709    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:49.254214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.253723    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:49.254214 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.253749    2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-149.ec2.internal"
Jul 28 12:53:49.261713 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.261696    2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-149.ec2.internal"
Jul 28 12:53:49.261763 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.261720    2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-149.ec2.internal\": node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:49.275504 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.275447    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:53:49.276614 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.276598    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:53:49.276708 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.276629    2553 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:53:49.276708 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.276650    2553 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:53:49.276708 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.276683    2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:53:49.277410 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:49.277396    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:49.278294 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:49.278271    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:50.570959 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.570927    2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"]
Jul 28 12:53:50.571099 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.571023    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:50.571099 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:50.570944    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:50.571896 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.571881    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:50.571964 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.571911    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:50.571964 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.571924    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:50.573783 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.573771    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:50.573942 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.573929    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.574003 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.573958    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:50.574454 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.574434    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:50.574517 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.574462    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:50.574517 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.574440    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:50.574517 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.574492    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:50.574517 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.574502    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:50.574517 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.574471    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:50.576392 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.576378    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.576439 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.576408    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:50.577014 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.577000    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:50.577092 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.577030    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:50.577092 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.577062    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:50.586976 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.586955    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae6a3c6932db4acd7fa36b4d61252efe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal\" (UID: \"ae6a3c6932db4acd7fa36b4d61252efe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.587043 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.586983    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f5159b379a705575dacfde81933eecbe-config\") pod \"kube-apiserver-proxy-ip-10-0-135-149.ec2.internal\" (UID: \"f5159b379a705575dacfde81933eecbe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.587043 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.586999    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae6a3c6932db4acd7fa36b4d61252efe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal\" (UID: \"ae6a3c6932db4acd7fa36b4d61252efe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.615405 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:50.615385    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-149.ec2.internal\" not found" node="ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.619569 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:50.619552    2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-149.ec2.internal\" not found" node="ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.671600 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:50.671580    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:50.687136 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.687113    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae6a3c6932db4acd7fa36b4d61252efe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal\" (UID: \"ae6a3c6932db4acd7fa36b4d61252efe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.687217 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.687168    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae6a3c6932db4acd7fa36b4d61252efe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal\" (UID: \"ae6a3c6932db4acd7fa36b4d61252efe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.687217 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.687199    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae6a3c6932db4acd7fa36b4d61252efe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal\" (UID: \"ae6a3c6932db4acd7fa36b4d61252efe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.687286 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.687235    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f5159b379a705575dacfde81933eecbe-config\") pod \"kube-apiserver-proxy-ip-10-0-135-149.ec2.internal\" (UID: \"f5159b379a705575dacfde81933eecbe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.687286 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.687251    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae6a3c6932db4acd7fa36b4d61252efe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal\" (UID: \"ae6a3c6932db4acd7fa36b4d61252efe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.687286 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.687271    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f5159b379a705575dacfde81933eecbe-config\") pod \"kube-apiserver-proxy-ip-10-0-135-149.ec2.internal\" (UID: \"f5159b379a705575dacfde81933eecbe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.772273 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:50.772205    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:50.872716 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:50.872681    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:50.918168 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.918146    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.922816 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:50.922704    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:50.973600 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:50.973569    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:51.074137 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:51.074047    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:51.174581 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:51.174553    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:51.186870 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.186851    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:51.228863 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.228837    2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:53:51.229256 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.228960    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:53:51.229256 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.229002    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:53:51.275274 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:51.275242    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:51.298806 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.298755    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:48:50 +0000 UTC" deadline="2028-03-29 05:17:46.005564131 +0000 UTC"
Jul 28 12:53:51.298806 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.298781    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14632h23m54.706785745s"
Jul 28 12:53:51.375551 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:51.375456    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-149.ec2.internal\" not found"
Jul 28 12:53:51.395702 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.395673    2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:51.485848 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.485819    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:51.499164 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.499139    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:53:51.500638 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.500626    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"
Jul 28 12:53:51.512084 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.512063    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:53:51.688783 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:51.688737    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6a3c6932db4acd7fa36b4d61252efe.slice/crio-c3c3c9daf9ab20d53d24c2f2734f71a55d7aae4c48753cf7835114ce6a627075 WatchSource:0}: Error finding container c3c3c9daf9ab20d53d24c2f2734f71a55d7aae4c48753cf7835114ce6a627075: Status 404 returned error can't find the container with id c3c3c9daf9ab20d53d24c2f2734f71a55d7aae4c48753cf7835114ce6a627075
Jul 28 12:53:51.689205 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:51.689184    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5159b379a705575dacfde81933eecbe.slice/crio-d08c2f78146566da0bf68659132e10300bec010590d94f2b0571e86e0f5fae7a WatchSource:0}: Error finding container d08c2f78146566da0bf68659132e10300bec010590d94f2b0571e86e0f5fae7a: Status 404 returned error can't find the container with id d08c2f78146566da0bf68659132e10300bec010590d94f2b0571e86e0f5fae7a
Jul 28 12:53:51.693722 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.693699    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:53:51.843446 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:51.843413    2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:52.263341 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.263311    2553 apiserver.go:52] "Watching apiserver"
Jul 28 12:53:52.273884 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.273855    2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:53:52.275521 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.275496    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-jf286","openshift-network-diagnostics/network-check-target-cqxxn","openshift-network-operator/iptables-alerter-tkkwh","openshift-ovn-kubernetes/ovnkube-node-8hmhh","kube-system/global-pull-secret-syncer-2f4qz","kube-system/konnectivity-agent-n6sj9","kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5","openshift-dns/node-resolver-w497j","openshift-multus/multus-additional-cni-plugins-5wdsp","openshift-multus/network-metrics-daemon-qrq22","openshift-cluster-node-tuning-operator/tuned-pr9jm","openshift-image-registry/node-ca-mzt5m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal"]
Jul 28 12:53:52.278043 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.278020    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.280351 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.280328    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.280749 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.280726    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:53:52.280986 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.280962    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:53:52.281094 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.281077    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:53:52.281304 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.281284    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5mf46\""
Jul 28 12:53:52.281382 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.281327    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:53:52.281382 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.281353    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:53:52.282475 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.282456    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.283012 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.282993    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:53:52.283422 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.283404    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dqrr6\""
Jul 28 12:53:52.283513 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.283453    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:53:52.283779 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.283760    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:53:52.284973 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.284952    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:53:52.285061 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.284980    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.285382 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.285360    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:53:52.285607 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.285589    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:53:52.285701 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.285685    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4dnwb\""
Jul 28 12:53:52.285886 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.285593    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:53:52.287439 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.287422    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.287541 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.287510    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:53:52.287935 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.287874    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:53:52.289585 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.289101    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:53:52.289585 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.289135    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:53:52.289585 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.289225    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wt2rq\""
Jul 28 12:53:52.289585 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.289281    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:53:52.289585 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.289327    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:53:52.289861 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.289710    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:53:52.291833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.291551    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:53:52.294142 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.294107    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.294237 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.294193    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:53:52.294335 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.294316    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mm6mg\""
Jul 28 12:53:52.294433 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.294419    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:53:52.295634 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295616    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-system-cni-dir\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.295730 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295644    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-os-release\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.295730 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295669    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14061f83-1fc2-4769-8f1a-ebd040d3e4e6-host-slash\") pod \"iptables-alerter-tkkwh\" (UID: \"14061f83-1fc2-4769-8f1a-ebd040d3e4e6\") " pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.295730 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295692    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-run-netns\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.295962 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295726    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-run-systemd\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.295962 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295764    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-var-lib-openvswitch\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.295962 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295789    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-node-log\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.295962 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295826    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51477dbb-d4b7-448e-9583-64f2d909364d-ovn-node-metrics-cert\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.295962 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295857    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.295962 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295880    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2439221-c604-428e-885e-84788d713251-serviceca\") pod \"node-ca-mzt5m\" (UID: \"c2439221-c604-428e-885e-84788d713251\") " pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.295962 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295920    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-cni-bin\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.295962 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295941    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-cnibin\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.295976    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4sw\" (UniqueName: \"kubernetes.io/projected/14d239a1-4dfe-4102-b8e5-ddc411b60264-kube-api-access-nr4sw\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296008    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-448qr\" (UniqueName: \"kubernetes.io/projected/c2439221-c604-428e-885e-84788d713251-kube-api-access-448qr\") pod \"node-ca-mzt5m\" (UID: \"c2439221-c604-428e-885e-84788d713251\") " pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296032    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14061f83-1fc2-4769-8f1a-ebd040d3e4e6-iptables-alerter-script\") pod \"iptables-alerter-tkkwh\" (UID: \"14061f83-1fc2-4769-8f1a-ebd040d3e4e6\") " pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296055    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51477dbb-d4b7-448e-9583-64f2d909364d-env-overrides\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296077    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jzc\" (UniqueName: \"kubernetes.io/projected/51477dbb-d4b7-448e-9583-64f2d909364d-kube-api-access-d6jzc\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296129    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2058fd10-4ec4-45cd-b41f-3496159fa93c-kubelet-config\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296173    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2058fd10-4ec4-45cd-b41f-3496159fa93c-dbus\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296212    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2439221-c604-428e-885e-84788d713251-host\") pod \"node-ca-mzt5m\" (UID: \"c2439221-c604-428e-885e-84788d713251\") " pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296241    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-etc-openvswitch\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296265    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-run-openvswitch\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296313    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-run-ovn\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296345    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.296370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296372    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51477dbb-d4b7-448e-9583-64f2d909364d-ovnkube-config\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296406    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51477dbb-d4b7-448e-9583-64f2d909364d-ovnkube-script-lib\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296456    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frl9c\" (UniqueName: \"kubernetes.io/projected/14061f83-1fc2-4769-8f1a-ebd040d3e4e6-kube-api-access-frl9c\") pod \"iptables-alerter-tkkwh\" (UID: \"14061f83-1fc2-4769-8f1a-ebd040d3e4e6\") " pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296532    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296509    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-cni-netd\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296586    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-kubelet\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296601    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-log-socket\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296619    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14d239a1-4dfe-4102-b8e5-ddc411b60264-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296634    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-systemd-units\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296647    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-slash\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296661    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296675    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296688    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14d239a1-4dfe-4102-b8e5-ddc411b60264-cni-binary-copy\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296708    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14d239a1-4dfe-4102-b8e5-ddc411b60264-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296982    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:53:52.297045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.296984    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rsznk\""
Jul 28 12:53:52.297819 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.297137    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:53:52.298712 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.298691    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:53:52.298823 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.298770    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:53:52.299158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.299140    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:53:52.299339 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.299326    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:53:52.299561 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.299544    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dmsjd\""
Jul 28 12:53:52.299905 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.299829    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:53:52.300996 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.300978    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.303349 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.303331    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:53:52.303995 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.303977    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-trbw7\""
Jul 28 12:53:52.304373 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.304357    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:52.304463 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.304438    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:53:52.304706 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.304687    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:53:52.307104 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.307078    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.309511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.309462    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r4mt6\""
Jul 28 12:53:52.310054 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.310035    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:53:52.310168 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.310074    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:53:52.321212 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.321194    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g92r5"
Jul 28 12:53:52.328849 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.328827    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g92r5"
Jul 28 12:53:52.386847 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.386815    2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:53:52.397647 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397608    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-sys-fs\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.397819 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397652    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdns\" (UniqueName: \"kubernetes.io/projected/7baa7a60-a59d-4131-821e-aaa0a2d16218-kube-api-access-5hdns\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.397819 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397688    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-os-release\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.397819 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397718    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14061f83-1fc2-4769-8f1a-ebd040d3e4e6-host-slash\") pod \"iptables-alerter-tkkwh\" (UID: \"14061f83-1fc2-4769-8f1a-ebd040d3e4e6\") " pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.397819 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397740    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-run-systemd\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.397819 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397757    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-node-log\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.397819 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397771    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-os-release\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397822    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-run-systemd\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397826    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14061f83-1fc2-4769-8f1a-ebd040d3e4e6-host-slash\") pod \"iptables-alerter-tkkwh\" (UID: \"14061f83-1fc2-4769-8f1a-ebd040d3e4e6\") " pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397781    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51477dbb-d4b7-448e-9583-64f2d909364d-ovn-node-metrics-cert\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397842    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-node-log\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397874    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397898    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-system-cni-dir\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397931    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-conf-dir\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.397979    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2439221-c604-428e-885e-84788d713251-serviceca\") pod \"node-ca-mzt5m\" (UID: \"c2439221-c604-428e-885e-84788d713251\") " pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398005    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-cni-bin\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398032    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqgw\" (UniqueName: \"kubernetes.io/projected/1bece3c2-4eb4-4219-a757-45a57c351842-kube-api-access-5mqgw\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398059    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-run-netns\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.398078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398082    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-var-lib-cni-bin\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398108    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-kubernetes\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398130    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-cni-bin\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398126    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398153    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-var-lib-kubelet\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398171    2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398177    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-var-lib-cni-multus\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398242    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4sw\" (UniqueName: \"kubernetes.io/projected/14d239a1-4dfe-4102-b8e5-ddc411b60264-kube-api-access-nr4sw\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2058fd10-4ec4-45cd-b41f-3496159fa93c-kubelet-config\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398293    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2058fd10-4ec4-45cd-b41f-3496159fa93c-dbus\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398320    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xgkk\" (UniqueName: \"kubernetes.io/projected/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-kube-api-access-4xgkk\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398345    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-lib-modules\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398368    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398372    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2058fd10-4ec4-45cd-b41f-3496159fa93c-kubelet-config\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398412    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398416    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2439221-c604-428e-885e-84788d713251-serviceca\") pod \"node-ca-mzt5m\" (UID: \"c2439221-c604-428e-885e-84788d713251\") " pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398440    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-etc-openvswitch\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.398663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398469    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-run-openvswitch\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398508    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-etc-openvswitch\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398513    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-run-openvswitch\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398538    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-run-ovn\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398567    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398596    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51477dbb-d4b7-448e-9583-64f2d909364d-ovnkube-config\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398609    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-run-ovn\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398568    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2058fd10-4ec4-45cd-b41f-3496159fa93c-dbus\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398620    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-tuned\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398636    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398656    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-device-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398679    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-cni-netd\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398701    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-daemon-config\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398723    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-sysctl-conf\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398745    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-run\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398754    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-cni-netd\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398808    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302c2c0a-0dd0-4af7-83fb-f52dd09da666-tmp-dir\") pod \"node-resolver-w497j\" (UID: \"302c2c0a-0dd0-4af7-83fb-f52dd09da666\") " pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.399562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398834    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-run-k8s-cni-cncf-io\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398858    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-sysctl-d\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398882    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-socket-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398907    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-registration-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398933    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-systemd-units\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398958    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398980    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-systemd\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.398984    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-systemd-units\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399002    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89cc2b6b-5922-4400-8607-99a5ff7485f0-tmp\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399024    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxcjz\" (UniqueName: \"kubernetes.io/projected/89cc2b6b-5922-4400-8607-99a5ff7485f0-kube-api-access-bxcjz\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399050    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14d239a1-4dfe-4102-b8e5-ddc411b60264-cni-binary-copy\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.399071    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399077    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14d239a1-4dfe-4102-b8e5-ddc411b60264-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399101    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcsrg\" (UniqueName: \"kubernetes.io/projected/302c2c0a-0dd0-4af7-83fb-f52dd09da666-kube-api-access-gcsrg\") pod \"node-resolver-w497j\" (UID: \"302c2c0a-0dd0-4af7-83fb-f52dd09da666\") " pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399129    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51477dbb-d4b7-448e-9583-64f2d909364d-ovnkube-config\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399139    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-cnibin\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.400346 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.399189    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret podName:2058fd10-4ec4-45cd-b41f-3496159fa93c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:52.899151012 +0000 UTC m=+2.936519903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret") pod "global-pull-secret-syncer-2f4qz" (UID: "2058fd10-4ec4-45cd-b41f-3496159fa93c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399210    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-os-release\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399234    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-cni-binary-copy\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399255    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-socket-dir-parent\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399279    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-modprobe-d\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399304    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-system-cni-dir\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399330    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-run-netns\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399355    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-var-lib-openvswitch\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399392    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba0380da-f7c1-46d2-beca-b5475dcc58bb-agent-certs\") pod \"konnectivity-agent-n6sj9\" (UID: \"ba0380da-f7c1-46d2-beca-b5475dcc58bb\") " pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399428    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-cnibin\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399442    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-system-cni-dir\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399400    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-var-lib-openvswitch\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399458    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-448qr\" (UniqueName: \"kubernetes.io/projected/c2439221-c604-428e-885e-84788d713251-kube-api-access-448qr\") pod \"node-ca-mzt5m\" (UID: \"c2439221-c604-428e-885e-84788d713251\") " pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399483    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14061f83-1fc2-4769-8f1a-ebd040d3e4e6-iptables-alerter-script\") pod \"iptables-alerter-tkkwh\" (UID: \"14061f83-1fc2-4769-8f1a-ebd040d3e4e6\") " pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399489    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-run-netns\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399513    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51477dbb-d4b7-448e-9583-64f2d909364d-env-overrides\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399635    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jzc\" (UniqueName: \"kubernetes.io/projected/51477dbb-d4b7-448e-9583-64f2d909364d-kube-api-access-d6jzc\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.401068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399660    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14d239a1-4dfe-4102-b8e5-ddc411b60264-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399677    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14d239a1-4dfe-4102-b8e5-ddc411b60264-cni-binary-copy\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399664    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-sysconfig\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399776    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14d239a1-4dfe-4102-b8e5-ddc411b60264-cnibin\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399828    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-host\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399855    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2439221-c604-428e-885e-84788d713251-host\") pod \"node-ca-mzt5m\" (UID: \"c2439221-c604-428e-885e-84788d713251\") " pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399892    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51477dbb-d4b7-448e-9583-64f2d909364d-ovnkube-script-lib\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399918    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba0380da-f7c1-46d2-beca-b5475dcc58bb-konnectivity-ca\") pod \"konnectivity-agent-n6sj9\" (UID: \"ba0380da-f7c1-46d2-beca-b5475dcc58bb\") " pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399943    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frl9c\" (UniqueName: \"kubernetes.io/projected/14061f83-1fc2-4769-8f1a-ebd040d3e4e6-kube-api-access-frl9c\") pod \"iptables-alerter-tkkwh\" (UID: \"14061f83-1fc2-4769-8f1a-ebd040d3e4e6\") " pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.399967    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-hostroot\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400023    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51477dbb-d4b7-448e-9583-64f2d909364d-env-overrides\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400038    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14061f83-1fc2-4769-8f1a-ebd040d3e4e6-iptables-alerter-script\") pod \"iptables-alerter-tkkwh\" (UID: \"14061f83-1fc2-4769-8f1a-ebd040d3e4e6\") " pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400102    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-run-multus-certs\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400128    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2439221-c604-428e-885e-84788d713251-host\") pod \"node-ca-mzt5m\" (UID: \"c2439221-c604-428e-885e-84788d713251\") " pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400164    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-etc-selinux\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400199    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-kubelet\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400225    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-cni-dir\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.401867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400261    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-kubelet\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400295    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-var-lib-kubelet\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400320    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-etc-kubernetes\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400348    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-log-socket\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400377    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14d239a1-4dfe-4102-b8e5-ddc411b60264-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400392    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-log-socket\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400405    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-slash\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400436    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400441    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51477dbb-d4b7-448e-9583-64f2d909364d-ovnkube-script-lib\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400443    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-slash\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400462    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/302c2c0a-0dd0-4af7-83fb-f52dd09da666-hosts-file\") pod \"node-resolver-w497j\" (UID: \"302c2c0a-0dd0-4af7-83fb-f52dd09da666\") " pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400488    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51477dbb-d4b7-448e-9583-64f2d909364d-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400489    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-sys\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.400853    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14d239a1-4dfe-4102-b8e5-ddc411b60264-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.402500 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.401696    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51477dbb-d4b7-448e-9583-64f2d909364d-ovn-node-metrics-cert\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.414624 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.414594    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jzc\" (UniqueName: \"kubernetes.io/projected/51477dbb-d4b7-448e-9583-64f2d909364d-kube-api-access-d6jzc\") pod \"ovnkube-node-8hmhh\" (UID: \"51477dbb-d4b7-448e-9583-64f2d909364d\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.414763 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.414654    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-448qr\" (UniqueName: \"kubernetes.io/projected/c2439221-c604-428e-885e-84788d713251-kube-api-access-448qr\") pod \"node-ca-mzt5m\" (UID: \"c2439221-c604-428e-885e-84788d713251\") " pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.414899 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.414879    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4sw\" (UniqueName: \"kubernetes.io/projected/14d239a1-4dfe-4102-b8e5-ddc411b60264-kube-api-access-nr4sw\") pod \"multus-additional-cni-plugins-5wdsp\" (UID: \"14d239a1-4dfe-4102-b8e5-ddc411b60264\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.415472 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.415454    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frl9c\" (UniqueName: \"kubernetes.io/projected/14061f83-1fc2-4769-8f1a-ebd040d3e4e6-kube-api-access-frl9c\") pod \"iptables-alerter-tkkwh\" (UID: \"14061f83-1fc2-4769-8f1a-ebd040d3e4e6\") " pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.475245 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.475188    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal" event={"ID":"f5159b379a705575dacfde81933eecbe","Type":"ContainerStarted","Data":"d08c2f78146566da0bf68659132e10300bec010590d94f2b0571e86e0f5fae7a"}
Jul 28 12:53:52.476816 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.476770    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal" event={"ID":"ae6a3c6932db4acd7fa36b4d61252efe","Type":"ContainerStarted","Data":"c3c3c9daf9ab20d53d24c2f2734f71a55d7aae4c48753cf7835114ce6a627075"}
Jul 28 12:53:52.500803 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500763    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba0380da-f7c1-46d2-beca-b5475dcc58bb-agent-certs\") pod \"konnectivity-agent-n6sj9\" (UID: \"ba0380da-f7c1-46d2-beca-b5475dcc58bb\") " pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:53:52.500949 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500820    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-sysconfig\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.500949 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500847    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-host\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.500949 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500874    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba0380da-f7c1-46d2-beca-b5475dcc58bb-konnectivity-ca\") pod \"konnectivity-agent-n6sj9\" (UID: \"ba0380da-f7c1-46d2-beca-b5475dcc58bb\") " pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:53:52.500949 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500899    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-hostroot\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.500949 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500926    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-run-multus-certs\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500953    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-etc-selinux\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500964    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-sysconfig\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500983    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-hostroot\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500998    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-run-multus-certs\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500976    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-cni-dir\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501047    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-cni-dir\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501040    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-var-lib-kubelet\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.500964    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-host\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501086    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-etc-kubernetes\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501089    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-var-lib-kubelet\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501102    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-etc-selinux\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501117    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/302c2c0a-0dd0-4af7-83fb-f52dd09da666-hosts-file\") pod \"node-resolver-w497j\" (UID: \"302c2c0a-0dd0-4af7-83fb-f52dd09da666\") " pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501141    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-etc-kubernetes\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501140    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-sys\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.501199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501189    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-sys-fs\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501210    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/302c2c0a-0dd0-4af7-83fb-f52dd09da666-hosts-file\") pod \"node-resolver-w497j\" (UID: \"302c2c0a-0dd0-4af7-83fb-f52dd09da666\") " pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501248    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-sys-fs\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501248    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-sys\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501270    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdns\" (UniqueName: \"kubernetes.io/projected/7baa7a60-a59d-4131-821e-aaa0a2d16218-kube-api-access-5hdns\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501310    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501366    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-system-cni-dir\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501389    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-conf-dir\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501415    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqgw\" (UniqueName: \"kubernetes.io/projected/1bece3c2-4eb4-4219-a757-45a57c351842-kube-api-access-5mqgw\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501446    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-run-netns\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.501459    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501470    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-var-lib-cni-bin\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501472    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-conf-dir\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501496    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-kubernetes\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.501512    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs podName:1bece3c2-4eb4-4219-a757-45a57c351842 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:53.001495565 +0000 UTC m=+3.038864464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs") pod "network-metrics-daemon-qrq22" (UID: "1bece3c2-4eb4-4219-a757-45a57c351842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501521    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-system-cni-dir\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501542    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-kubernetes\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.501924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501541    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-var-lib-kubelet\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501562    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-run-netns\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501574    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-var-lib-cni-multus\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501585    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-var-lib-kubelet\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501600    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba0380da-f7c1-46d2-beca-b5475dcc58bb-konnectivity-ca\") pod \"konnectivity-agent-n6sj9\" (UID: \"ba0380da-f7c1-46d2-beca-b5475dcc58bb\") " pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501601    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xgkk\" (UniqueName: \"kubernetes.io/projected/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-kube-api-access-4xgkk\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501638    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-var-lib-cni-bin\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501644    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-lib-modules\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501651    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-var-lib-cni-multus\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501667    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501684    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501706    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-tuned\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501731    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-device-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501747    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501756    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-daemon-config\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501778    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-lib-modules\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501787    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-sysctl-conf\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.502718 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501838    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-run\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501861    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-device-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501868    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302c2c0a-0dd0-4af7-83fb-f52dd09da666-tmp-dir\") pod \"node-resolver-w497j\" (UID: \"302c2c0a-0dd0-4af7-83fb-f52dd09da666\") " pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501898    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-run-k8s-cni-cncf-io\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501922    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-sysctl-d\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501945    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-socket-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501970    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-registration-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.501976    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-host-run-k8s-cni-cncf-io\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502008    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-systemd\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502030    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89cc2b6b-5922-4400-8607-99a5ff7485f0-tmp\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502036    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-sysctl-d\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502052    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxcjz\" (UniqueName: \"kubernetes.io/projected/89cc2b6b-5922-4400-8607-99a5ff7485f0-kube-api-access-bxcjz\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502078    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-registration-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502103    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-sysctl-conf\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502111    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcsrg\" (UniqueName: \"kubernetes.io/projected/302c2c0a-0dd0-4af7-83fb-f52dd09da666-kube-api-access-gcsrg\") pod \"node-resolver-w497j\" (UID: \"302c2c0a-0dd0-4af7-83fb-f52dd09da666\") " pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502150    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-run\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502161    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/302c2c0a-0dd0-4af7-83fb-f52dd09da666-tmp-dir\") pod \"node-resolver-w497j\" (UID: \"302c2c0a-0dd0-4af7-83fb-f52dd09da666\") " pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502178    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7baa7a60-a59d-4131-821e-aaa0a2d16218-socket-dir\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.503451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502200    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-cnibin\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502238    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-systemd\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502138    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-cnibin\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502281    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-os-release\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502305    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-cni-binary-copy\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502331    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-socket-dir-parent\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502315    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-daemon-config\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502361    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-modprobe-d\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502380    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-os-release\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502452    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-modprobe-d\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502516    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-multus-socket-dir-parent\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.502877    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-cni-binary-copy\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.504080 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.503669    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba0380da-f7c1-46d2-beca-b5475dcc58bb-agent-certs\") pod \"konnectivity-agent-n6sj9\" (UID: \"ba0380da-f7c1-46d2-beca-b5475dcc58bb\") " pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:53:52.504495 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.504473    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89cc2b6b-5922-4400-8607-99a5ff7485f0-etc-tuned\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.504495 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.504490    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89cc2b6b-5922-4400-8607-99a5ff7485f0-tmp\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.519812 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.519677    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqgw\" (UniqueName: \"kubernetes.io/projected/1bece3c2-4eb4-4219-a757-45a57c351842-kube-api-access-5mqgw\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:52.520400 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.520046    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdns\" (UniqueName: \"kubernetes.io/projected/7baa7a60-a59d-4131-821e-aaa0a2d16218-kube-api-access-5hdns\") pod \"aws-ebs-csi-driver-node-cscp5\" (UID: \"7baa7a60-a59d-4131-821e-aaa0a2d16218\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.520536 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.520514    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:52.520596 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.520553    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:52.520596 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.520568    2553 projected.go:196] Error preparing data for projected volume kube-api-access-zbld7 for pod openshift-network-diagnostics/network-check-target-cqxxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:52.520694 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.520630    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7 podName:2ebe21a0-6999-409b-a63b-0fd0265d25b5 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:53.020611828 +0000 UTC m=+3.057980731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zbld7" (UniqueName: "kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7") pod "network-check-target-cqxxn" (UID: "2ebe21a0-6999-409b-a63b-0fd0265d25b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:52.521075 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.521054    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcsrg\" (UniqueName: \"kubernetes.io/projected/302c2c0a-0dd0-4af7-83fb-f52dd09da666-kube-api-access-gcsrg\") pod \"node-resolver-w497j\" (UID: \"302c2c0a-0dd0-4af7-83fb-f52dd09da666\") " pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.521302 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.521281    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xgkk\" (UniqueName: \"kubernetes.io/projected/812e57fd-70cf-4e3d-be5a-b2ad7fca3c21-kube-api-access-4xgkk\") pod \"multus-jf286\" (UID: \"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21\") " pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.521302 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.521294    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxcjz\" (UniqueName: \"kubernetes.io/projected/89cc2b6b-5922-4400-8607-99a5ff7485f0-kube-api-access-bxcjz\") pod \"tuned-pr9jm\" (UID: \"89cc2b6b-5922-4400-8607-99a5ff7485f0\") " pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.592429 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.592398    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5wdsp"
Jul 28 12:53:52.600906 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.600877    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzt5m"
Jul 28 12:53:52.608531 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.608512    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tkkwh"
Jul 28 12:53:52.614184 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.614165    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:53:52.619808 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.619771    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:53:52.627351 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.627330    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w497j"
Jul 28 12:53:52.634002 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.633983    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5"
Jul 28 12:53:52.640590 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.640573    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jf286"
Jul 28 12:53:52.646138 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.646122    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pr9jm"
Jul 28 12:53:52.667312 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.667289    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:52.906052 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:52.906022    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:52.906258 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.906155    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:52.906258 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:52.906218    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret podName:2058fd10-4ec4-45cd-b41f-3496159fa93c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:53.906199967 +0000 UTC m=+3.943568868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret") pod "global-pull-secret-syncer-2f4qz" (UID: "2058fd10-4ec4-45cd-b41f-3496159fa93c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:52.907683 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:52.907660    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0380da_f7c1_46d2_beca_b5475dcc58bb.slice/crio-6382c2b669b4bf89c1e12be0dcd48a24f4927c9a0e0cb82c48857b7c367de59d WatchSource:0}: Error finding container 6382c2b669b4bf89c1e12be0dcd48a24f4927c9a0e0cb82c48857b7c367de59d: Status 404 returned error can't find the container with id 6382c2b669b4bf89c1e12be0dcd48a24f4927c9a0e0cb82c48857b7c367de59d
Jul 28 12:53:52.908137 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:52.908118    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2439221_c604_428e_885e_84788d713251.slice/crio-877f3a8df0afad8faef249531cd44038a0023e49f46c0a46009efcb5a21fa3b6 WatchSource:0}: Error finding container 877f3a8df0afad8faef249531cd44038a0023e49f46c0a46009efcb5a21fa3b6: Status 404 returned error can't find the container with id 877f3a8df0afad8faef249531cd44038a0023e49f46c0a46009efcb5a21fa3b6
Jul 28 12:53:52.912501 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:52.912469    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cc2b6b_5922_4400_8607_99a5ff7485f0.slice/crio-77a5dde564b2e28d5283b23c51e8b7ac655e906ec71c0cb825ec8105fb8a99f6 WatchSource:0}: Error finding container 77a5dde564b2e28d5283b23c51e8b7ac655e906ec71c0cb825ec8105fb8a99f6: Status 404 returned error can't find the container with id 77a5dde564b2e28d5283b23c51e8b7ac655e906ec71c0cb825ec8105fb8a99f6
Jul 28 12:53:52.916241 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:52.916220    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d239a1_4dfe_4102_b8e5_ddc411b60264.slice/crio-8d8e87c1f523d2b47a8ce9d4206aac2f26b5082647f5af0c448facbbcdbf694b WatchSource:0}: Error finding container 8d8e87c1f523d2b47a8ce9d4206aac2f26b5082647f5af0c448facbbcdbf694b: Status 404 returned error can't find the container with id 8d8e87c1f523d2b47a8ce9d4206aac2f26b5082647f5af0c448facbbcdbf694b
Jul 28 12:53:52.916829 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:52.916768    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7baa7a60_a59d_4131_821e_aaa0a2d16218.slice/crio-19e981a8dfa4356d8a48e80778aae4938bbf8ef599fdfed453e0c99da3ae2405 WatchSource:0}: Error finding container 19e981a8dfa4356d8a48e80778aae4938bbf8ef599fdfed453e0c99da3ae2405: Status 404 returned error can't find the container with id 19e981a8dfa4356d8a48e80778aae4938bbf8ef599fdfed453e0c99da3ae2405
Jul 28 12:53:52.917624 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:52.917600    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302c2c0a_0dd0_4af7_83fb_f52dd09da666.slice/crio-e14de3d94d13994083f7635f507739b4ebb9897929c0a4756ee91b72873bea41 WatchSource:0}: Error finding container e14de3d94d13994083f7635f507739b4ebb9897929c0a4756ee91b72873bea41: Status 404 returned error can't find the container with id e14de3d94d13994083f7635f507739b4ebb9897929c0a4756ee91b72873bea41
Jul 28 12:53:52.918531 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:52.917768    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51477dbb_d4b7_448e_9583_64f2d909364d.slice/crio-1c262f26452ab366ca8688b0eafd84343a4cfccd618cc6f1348a239c830d060e WatchSource:0}: Error finding container 1c262f26452ab366ca8688b0eafd84343a4cfccd618cc6f1348a239c830d060e: Status 404 returned error can't find the container with id 1c262f26452ab366ca8688b0eafd84343a4cfccd618cc6f1348a239c830d060e
Jul 28 12:53:52.937774 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:52.937710    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812e57fd_70cf_4e3d_be5a_b2ad7fca3c21.slice/crio-a669b8f3b02abcf9fe94713d3ad973a781aa839f5a08856bf87fa852a5dcd415 WatchSource:0}: Error finding container a669b8f3b02abcf9fe94713d3ad973a781aa839f5a08856bf87fa852a5dcd415: Status 404 returned error can't find the container with id a669b8f3b02abcf9fe94713d3ad973a781aa839f5a08856bf87fa852a5dcd415
Jul 28 12:53:52.938474 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:53:52.938363    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14061f83_1fc2_4769_8f1a_ebd040d3e4e6.slice/crio-3b18c8e81da40b2df1999bb2878920839301e813c9eff8c2ca60f08ced436504 WatchSource:0}: Error finding container 3b18c8e81da40b2df1999bb2878920839301e813c9eff8c2ca60f08ced436504: Status 404 returned error can't find the container with id 3b18c8e81da40b2df1999bb2878920839301e813c9eff8c2ca60f08ced436504
Jul 28 12:53:53.007292 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.007263    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:53.007425 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.007372    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:53.007489 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.007430    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs podName:1bece3c2-4eb4-4219-a757-45a57c351842 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:54.007412901 +0000 UTC m=+4.044781790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs") pod "network-metrics-daemon-qrq22" (UID: "1bece3c2-4eb4-4219-a757-45a57c351842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:53.108040 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.108015    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:53:53.108405 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.108189    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:53.108405 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.108209    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:53.108405 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.108218    2553 projected.go:196] Error preparing data for projected volume kube-api-access-zbld7 for pod openshift-network-diagnostics/network-check-target-cqxxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:53.108405 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.108269    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7 podName:2ebe21a0-6999-409b-a63b-0fd0265d25b5 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:54.108253259 +0000 UTC m=+4.145622158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zbld7" (UniqueName: "kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7") pod "network-check-target-cqxxn" (UID: "2ebe21a0-6999-409b-a63b-0fd0265d25b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:53.332284 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.332213    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:48:52 +0000 UTC" deadline="2028-01-04 23:01:27.984441675 +0000 UTC"
Jul 28 12:53:53.332284 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.332277    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12610h7m34.652168498s"
Jul 28 12:53:53.470649 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.470573    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:53.470856 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.470713    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:53:53.471317 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.471142    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:53.471317 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.471262    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:53:53.486251 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.486216    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jf286" event={"ID":"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21","Type":"ContainerStarted","Data":"a669b8f3b02abcf9fe94713d3ad973a781aa839f5a08856bf87fa852a5dcd415"}
Jul 28 12:53:53.490846 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.490808    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" event={"ID":"51477dbb-d4b7-448e-9583-64f2d909364d","Type":"ContainerStarted","Data":"1c262f26452ab366ca8688b0eafd84343a4cfccd618cc6f1348a239c830d060e"}
Jul 28 12:53:53.496398 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.496371    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w497j" event={"ID":"302c2c0a-0dd0-4af7-83fb-f52dd09da666","Type":"ContainerStarted","Data":"e14de3d94d13994083f7635f507739b4ebb9897929c0a4756ee91b72873bea41"}
Jul 28 12:53:53.501632 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.501556    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pr9jm" event={"ID":"89cc2b6b-5922-4400-8607-99a5ff7485f0","Type":"ContainerStarted","Data":"77a5dde564b2e28d5283b23c51e8b7ac655e906ec71c0cb825ec8105fb8a99f6"}
Jul 28 12:53:53.510513 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.509855    2553 generic.go:356] "Generic (PLEG): container finished" podID="ae6a3c6932db4acd7fa36b4d61252efe" containerID="540ad6f29506978d654426af0eed8f5c402e5f5a03ee0d1562fb90c0a80bf6f1" exitCode=0
Jul 28 12:53:53.510513 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.509938    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal" event={"ID":"ae6a3c6932db4acd7fa36b4d61252efe","Type":"ContainerDied","Data":"540ad6f29506978d654426af0eed8f5c402e5f5a03ee0d1562fb90c0a80bf6f1"}
Jul 28 12:53:53.518990 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.518964    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tkkwh" event={"ID":"14061f83-1fc2-4769-8f1a-ebd040d3e4e6","Type":"ContainerStarted","Data":"3b18c8e81da40b2df1999bb2878920839301e813c9eff8c2ca60f08ced436504"}
Jul 28 12:53:53.534810 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.534771    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" event={"ID":"14d239a1-4dfe-4102-b8e5-ddc411b60264","Type":"ContainerStarted","Data":"8d8e87c1f523d2b47a8ce9d4206aac2f26b5082647f5af0c448facbbcdbf694b"}
Jul 28 12:53:53.548318 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.548286    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5" event={"ID":"7baa7a60-a59d-4131-821e-aaa0a2d16218","Type":"ContainerStarted","Data":"19e981a8dfa4356d8a48e80778aae4938bbf8ef599fdfed453e0c99da3ae2405"}
Jul 28 12:53:53.554137 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.554105    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzt5m" event={"ID":"c2439221-c604-428e-885e-84788d713251","Type":"ContainerStarted","Data":"877f3a8df0afad8faef249531cd44038a0023e49f46c0a46009efcb5a21fa3b6"}
Jul 28 12:53:53.567937 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.567871    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n6sj9" event={"ID":"ba0380da-f7c1-46d2-beca-b5475dcc58bb","Type":"ContainerStarted","Data":"6382c2b669b4bf89c1e12be0dcd48a24f4927c9a0e0cb82c48857b7c367de59d"}
Jul 28 12:53:53.730923 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.730899    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:53.913617 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:53.913577    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:53.913863 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.913844    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:53.913955 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:53.913919    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret podName:2058fd10-4ec4-45cd-b41f-3496159fa93c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:55.913900335 +0000 UTC m=+5.951269232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret") pod "global-pull-secret-syncer-2f4qz" (UID: "2058fd10-4ec4-45cd-b41f-3496159fa93c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:54.014412 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:54.014331    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:54.014569 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:54.014474    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:54.014569 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:54.014538    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs podName:1bece3c2-4eb4-4219-a757-45a57c351842 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:56.014519008 +0000 UTC m=+6.051887911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs") pod "network-metrics-daemon-qrq22" (UID: "1bece3c2-4eb4-4219-a757-45a57c351842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:54.115728 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:54.115690    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:53:54.115934 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:54.115917    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:54.116008 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:54.115951    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:54.116008 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:54.115973    2553 projected.go:196] Error preparing data for projected volume kube-api-access-zbld7 for pod openshift-network-diagnostics/network-check-target-cqxxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:54.116110 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:54.116043    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7 podName:2ebe21a0-6999-409b-a63b-0fd0265d25b5 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:56.116024334 +0000 UTC m=+6.153393232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zbld7" (UniqueName: "kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7") pod "network-check-target-cqxxn" (UID: "2ebe21a0-6999-409b-a63b-0fd0265d25b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:54.333494 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:54.333400    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:48:52 +0000 UTC" deadline="2028-04-24 12:39:56.141589494 +0000 UTC"
Jul 28 12:53:54.333494 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:54.333443    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15263h46m1.80815059s"
Jul 28 12:53:54.473704 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:54.471216    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:53:54.473704 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:54.471340    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:53:54.589725 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:54.589634    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal" event={"ID":"ae6a3c6932db4acd7fa36b4d61252efe","Type":"ContainerStarted","Data":"d35270787834f2485c7cdb8430009077d9e88d04c9f29c7ce22e4a230542d3c1"}
Jul 28 12:53:54.606666 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:54.606607    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-149.ec2.internal" podStartSLOduration=3.60658842 podStartE2EDuration="3.60658842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:54.605963343 +0000 UTC m=+4.643332250" watchObservedRunningTime="2026-07-28 12:53:54.60658842 +0000 UTC m=+4.643957329"
Jul 28 12:53:55.471139 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:55.470981    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:55.471139 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:55.471121    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:53:55.471632 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:55.471198    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:55.471632 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:55.471280    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:53:55.934258 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:55.934174    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:55.934423 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:55.934332    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:55.934423 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:55.934392    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret podName:2058fd10-4ec4-45cd-b41f-3496159fa93c nodeName:}" failed. No retries permitted until 2026-07-28 12:53:59.934375028 +0000 UTC m=+9.971743927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret") pod "global-pull-secret-syncer-2f4qz" (UID: "2058fd10-4ec4-45cd-b41f-3496159fa93c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:56.035055 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:56.035016    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:56.035231 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:56.035209    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:56.035293 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:56.035282    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs podName:1bece3c2-4eb4-4219-a757-45a57c351842 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:00.03526143 +0000 UTC m=+10.072630332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs") pod "network-metrics-daemon-qrq22" (UID: "1bece3c2-4eb4-4219-a757-45a57c351842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:56.136370 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:56.136330    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:53:56.136562 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:56.136542    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:56.136630 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:56.136566    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:56.136630 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:56.136578    2553 projected.go:196] Error preparing data for projected volume kube-api-access-zbld7 for pod openshift-network-diagnostics/network-check-target-cqxxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:56.136728 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:56.136646    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7 podName:2ebe21a0-6999-409b-a63b-0fd0265d25b5 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:00.136627932 +0000 UTC m=+10.173996840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zbld7" (UniqueName: "kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7") pod "network-check-target-cqxxn" (UID: "2ebe21a0-6999-409b-a63b-0fd0265d25b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:56.473473 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:56.472989    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:53:56.473473 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:56.473114    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:53:57.471011 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:57.470973    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:57.471226 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:57.471018    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:57.471226 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:57.471134    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:53:57.471388 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:57.471249    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:53:58.471197 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:58.471103    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:53:58.471590 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:58.471238    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:53:59.470539 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:59.470502    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:59.470754 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:59.470650    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:53:59.470754 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:59.470709    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:53:59.470918 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:59.470841    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:53:59.971098 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:53:59.971062    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:53:59.971550 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:59.971203    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:59.971550 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:53:59.971273    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret podName:2058fd10-4ec4-45cd-b41f-3496159fa93c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:07.971254376 +0000 UTC m=+18.008623276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret") pod "global-pull-secret-syncer-2f4qz" (UID: "2058fd10-4ec4-45cd-b41f-3496159fa93c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:54:00.071431 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:00.071393    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:00.071617 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:00.071594    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:54:00.071698 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:00.071673    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs podName:1bece3c2-4eb4-4219-a757-45a57c351842 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:08.07165062 +0000 UTC m=+18.109019527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs") pod "network-metrics-daemon-qrq22" (UID: "1bece3c2-4eb4-4219-a757-45a57c351842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:54:00.172032 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:00.171956    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:00.172273 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:00.172110    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:54:00.172273 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:00.172138    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:54:00.172273 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:00.172152    2553 projected.go:196] Error preparing data for projected volume kube-api-access-zbld7 for pod openshift-network-diagnostics/network-check-target-cqxxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:54:00.172273 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:00.172219    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7 podName:2ebe21a0-6999-409b-a63b-0fd0265d25b5 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:08.172199365 +0000 UTC m=+18.209568255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zbld7" (UniqueName: "kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7") pod "network-check-target-cqxxn" (UID: "2ebe21a0-6999-409b-a63b-0fd0265d25b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:54:00.471662 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:00.471593    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:00.471847 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:00.471708    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:01.470784 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:01.470750    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:01.471233 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:01.470872    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:01.471233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:01.471048    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:01.471233 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:01.471148    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:02.471388 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:02.471347    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:02.471829 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:02.471487    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:03.471209 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:03.471169    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:03.471379 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:03.471169    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:03.471379 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:03.471288    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:03.471379 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:03.471349    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:04.471037 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:04.471005    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:04.471222 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:04.471114    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:05.471123 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:05.471092    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:05.471123 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:05.471131    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:05.471652 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:05.471242    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:05.471652 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:05.471328    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:06.470774 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:06.470739    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:06.470968 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:06.470874    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:07.470506 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:07.470465    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:07.471004 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:07.470512    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:07.471004 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:07.470623    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:07.471004 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:07.470732    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:08.028647 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:08.028606    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:08.028864 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:08.028808    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:54:08.028936 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:08.028885    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret podName:2058fd10-4ec4-45cd-b41f-3496159fa93c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:24.028864561 +0000 UTC m=+34.066233446 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret") pod "global-pull-secret-syncer-2f4qz" (UID: "2058fd10-4ec4-45cd-b41f-3496159fa93c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:54:08.129252 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:08.129210    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:08.129410 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:08.129330    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:54:08.129410 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:08.129394    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs podName:1bece3c2-4eb4-4219-a757-45a57c351842 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:24.129379132 +0000 UTC m=+34.166748017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs") pod "network-metrics-daemon-qrq22" (UID: "1bece3c2-4eb4-4219-a757-45a57c351842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:54:08.229875 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:08.229825    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:08.230039 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:08.230003    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:54:08.230039 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:08.230030    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:54:08.230138 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:08.230044    2553 projected.go:196] Error preparing data for projected volume kube-api-access-zbld7 for pod openshift-network-diagnostics/network-check-target-cqxxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:54:08.230138 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:08.230104    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7 podName:2ebe21a0-6999-409b-a63b-0fd0265d25b5 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:24.230090683 +0000 UTC m=+34.267459581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zbld7" (UniqueName: "kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7") pod "network-check-target-cqxxn" (UID: "2ebe21a0-6999-409b-a63b-0fd0265d25b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:54:08.470285 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:08.470244    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:08.470467 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:08.470373    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:09.471130 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:09.471090    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:09.471130 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:09.471116    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:09.471674 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:09.471259    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:09.471674 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:09.471404    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:10.471811 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:10.471768    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:10.472257 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:10.471887    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:11.470328 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.470300    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:11.470463 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:11.470438    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:11.470841 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.470821    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:11.470946 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:11.470920    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:11.628585 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.625846    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal" event={"ID":"f5159b379a705575dacfde81933eecbe","Type":"ContainerStarted","Data":"372e4a3bff3db11f029faf37ac499616a82e69690d3a92fe9a81389c6c540b33"}
Jul 28 12:54:11.631084 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.631052    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5" event={"ID":"7baa7a60-a59d-4131-821e-aaa0a2d16218","Type":"ContainerStarted","Data":"a4906f0911cdb7deffdb1838ae138c725731cf4caab5ca0ebb5e5a7faa059e1d"}
Jul 28 12:54:11.634676 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.633626    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzt5m" event={"ID":"c2439221-c604-428e-885e-84788d713251","Type":"ContainerStarted","Data":"19eaa7c0b8571aa9a061563fffaff652d304dd61ced479e114b16b6351a79c58"}
Jul 28 12:54:11.636603 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.636582    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n6sj9" event={"ID":"ba0380da-f7c1-46d2-beca-b5475dcc58bb","Type":"ContainerStarted","Data":"18ac37575d1f5e7aab4468754b976a38ac843919a2a058808cdfd85d80eed614"}
Jul 28 12:54:11.638867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.638777    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jf286" event={"ID":"812e57fd-70cf-4e3d-be5a-b2ad7fca3c21","Type":"ContainerStarted","Data":"e2f0762a8abb466dd9a9ec3cccd2688b4febe7d1bc9e8b060478c0616554883b"}
Jul 28 12:54:11.643212 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.641818    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" event={"ID":"51477dbb-d4b7-448e-9583-64f2d909364d","Type":"ContainerStarted","Data":"163848b83253512b32f343d5dcc91506ea97afd5411bd2dcdfcd4b42762d964f"}
Jul 28 12:54:11.643545 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.643505    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pr9jm" event={"ID":"89cc2b6b-5922-4400-8607-99a5ff7485f0","Type":"ContainerStarted","Data":"1b59fd471e151bacdb73b037b0e160224c0423bd62313101a3d677b432c417f1"}
Jul 28 12:54:11.651897 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.651644    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-149.ec2.internal" podStartSLOduration=20.651624082 podStartE2EDuration="20.651624082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:11.651116927 +0000 UTC m=+21.688485832" watchObservedRunningTime="2026-07-28 12:54:11.651624082 +0000 UTC m=+21.688992990"
Jul 28 12:54:11.680176 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.679616    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pr9jm" podStartSLOduration=3.235849309 podStartE2EDuration="21.679601021s" totalImagesPullingTime="18.443751712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:11.67914303 +0000 UTC m=+21.716511936" watchObservedRunningTime="2026-07-28 12:54:11.679601021 +0000 UTC m=+21.716969919"
Jul 28 12:54:11.704299 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.704245    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n6sj9" podStartSLOduration=3.258074069 podStartE2EDuration="21.704224138s" totalImagesPullingTime="18.446150069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:11.703431727 +0000 UTC m=+21.740800632" watchObservedRunningTime="2026-07-28 12:54:11.704224138 +0000 UTC m=+21.741593047"
Jul 28 12:54:11.727981 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.727898    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jf286" podStartSLOduration=3.168324773 podStartE2EDuration="21.727877081s" totalImagesPullingTime="18.559552308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:11.727664546 +0000 UTC m=+21.765033455" watchObservedRunningTime="2026-07-28 12:54:11.727877081 +0000 UTC m=+21.765245990"
Jul 28 12:54:11.743141 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.743116    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:11.743515 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.743471    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mzt5m" podStartSLOduration=3.319490656 podStartE2EDuration="21.743453153s" totalImagesPullingTime="18.423962497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:11.743327884 +0000 UTC m=+21.780696791" watchObservedRunningTime="2026-07-28 12:54:11.743453153 +0000 UTC m=+21.780822060"
Jul 28 12:54:11.818832 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.818801    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:11.917101 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:11.917076    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:12.027813 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.027772    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:12.128576 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.128533    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:12.234139 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.234111    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:12.297590 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.297493    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:12.470337 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.470307    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:12.470473 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:12.470430    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:12.648198 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.648161    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" event={"ID":"51477dbb-d4b7-448e-9583-64f2d909364d","Type":"ContainerStarted","Data":"6d72497e1b0afd598e260a2869d6be18e65bda212907550f1258f8d426121656"}
Jul 28 12:54:12.648616 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.648240    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" event={"ID":"51477dbb-d4b7-448e-9583-64f2d909364d","Type":"ContainerStarted","Data":"aeb12df74a2621346a0a313a9b33021000213455cdfcb0d99ce29118d84d3d61"}
Jul 28 12:54:12.648616 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.648275    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" event={"ID":"51477dbb-d4b7-448e-9583-64f2d909364d","Type":"ContainerStarted","Data":"66bef503faa1b96fbe332b1c0f264f190a57356c0ebaee19920a70026da56038"}
Jul 28 12:54:12.648616 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.648305    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" event={"ID":"51477dbb-d4b7-448e-9583-64f2d909364d","Type":"ContainerStarted","Data":"ef0806af6b872c2f9f565dc810aa69707931a15f7750d407fef06eddc6b64c26"}
Jul 28 12:54:12.648616 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.648340    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" event={"ID":"51477dbb-d4b7-448e-9583-64f2d909364d","Type":"ContainerStarted","Data":"b6aad0d4d798db55c7e35cd6e120f53c847c372cd9b5c84ec67283d7cd9f79a3"}
Jul 28 12:54:12.649610 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.649575    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w497j" event={"ID":"302c2c0a-0dd0-4af7-83fb-f52dd09da666","Type":"ContainerStarted","Data":"f70d8339a299e4e4515e23fce1a33e7b9596e669e984d7ed1f213f04fb3e74ed"}
Jul 28 12:54:12.651097 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.651070    2553 generic.go:356] "Generic (PLEG): container finished" podID="14d239a1-4dfe-4102-b8e5-ddc411b60264" containerID="06e21889328845c791c9c8b4041ff5f1a69aacde10cea70e242b69659356c164" exitCode=0
Jul 28 12:54:12.651233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.651164    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" event={"ID":"14d239a1-4dfe-4102-b8e5-ddc411b60264","Type":"ContainerDied","Data":"06e21889328845c791c9c8b4041ff5f1a69aacde10cea70e242b69659356c164"}
Jul 28 12:54:12.666958 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.666867    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w497j" podStartSLOduration=4.246675096 podStartE2EDuration="22.666847308s" totalImagesPullingTime="18.420172212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:12.665994018 +0000 UTC m=+22.703362924" watchObservedRunningTime="2026-07-28 12:54:12.666847308 +0000 UTC m=+22.704216215"
Jul 28 12:54:12.667088 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.666964    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:12.739035 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.739006    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:12.793258 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.793234    2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:54:12.839922 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.839883    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:12.946617 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:12.946583    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:13.047196 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.047161    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:13.357410 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.357241    2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:54:12.793254196Z","UUID":"a624aa76-5953-49a0-935e-824b4c816ed9","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:54:13.359029 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.359004    2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:54:13.359187 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.359046    2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:54:13.471148 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.471114    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:13.471148 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.471150    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:13.471456 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:13.471255    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:13.471456 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:13.471390    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:13.655686 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.655609    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tkkwh" event={"ID":"14061f83-1fc2-4769-8f1a-ebd040d3e4e6","Type":"ContainerStarted","Data":"b7ff903b175565a1cd96f9880452afba0de80018c704011e2f4dc42863aa4445"}
Jul 28 12:54:13.657560 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.657531    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5" event={"ID":"7baa7a60-a59d-4131-821e-aaa0a2d16218","Type":"ContainerStarted","Data":"c485aa740a61ed886796895647e084950d891c5dd62217ca8beaa5b88a23784a"}
Jul 28 12:54:13.674119 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.673865    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tkkwh" podStartSLOduration=5.257655127 podStartE2EDuration="23.673847761s" totalImagesPullingTime="18.416192634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:13.673765314 +0000 UTC m=+23.711134222" watchObservedRunningTime="2026-07-28 12:54:13.673847761 +0000 UTC m=+23.711216671"
Jul 28 12:54:13.683017 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.682994    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:54:13.683655 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.683630    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:54:13.874355 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.874306    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:13.953970 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:13.953931    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:14.229640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.229584    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w497j_302c2c0a-0dd0-4af7-83fb-f52dd09da666/dns-node-resolver/0.log"
Jul 28 12:54:14.427070 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.427019    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:14.475369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.475300    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:14.475557 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:14.475434    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:14.528177 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.528136    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:14.593980 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.593945    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:14.653946 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.653909    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:14.661584 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.661553    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5" event={"ID":"7baa7a60-a59d-4131-821e-aaa0a2d16218","Type":"ContainerStarted","Data":"6245fef1ba2f45ca2db5a2d6a5fa4e8dd608a44e8d1b12b9a3bfe2e3233a53f9"}
Jul 28 12:54:14.665493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.665465    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" event={"ID":"51477dbb-d4b7-448e-9583-64f2d909364d","Type":"ContainerStarted","Data":"9ce40012795d595493ed40192bd01d8d690e165334602ae5657ad9418e7eec76"}
Jul 28 12:54:14.665763 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.665704    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:54:14.666160 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.666123    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n6sj9"
Jul 28 12:54:14.685510 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.685469    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cscp5" podStartSLOduration=4.272393167 podStartE2EDuration="24.685456545s" totalImagesPullingTime="20.413063378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:14.685049359 +0000 UTC m=+24.722418282" watchObservedRunningTime="2026-07-28 12:54:14.685456545 +0000 UTC m=+24.722825452"
Jul 28 12:54:14.747280 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:14.747212    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:15.232608 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:15.232581    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzt5m_c2439221-c604-428e-885e-84788d713251/node-ca/0.log"
Jul 28 12:54:15.470526 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:15.470339    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:15.470688 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:15.470370    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:15.470688 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:15.470624    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:15.470760 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:15.470688    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:15.621486 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:15.621399    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:16.475088 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:16.475063    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:16.475479 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:16.475181    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:17.470866 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.470834    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:17.470866 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.470855    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:17.471090 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:17.470963    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:17.471090 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:17.471069    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:17.673759 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.673730    2553 generic.go:356] "Generic (PLEG): container finished" podID="14d239a1-4dfe-4102-b8e5-ddc411b60264" containerID="c22ebae849802bf78172cd881d17d58003f4269f0d080a8f2dd1ee3b5633d3b7" exitCode=0
Jul 28 12:54:17.674645 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.673822    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" event={"ID":"14d239a1-4dfe-4102-b8e5-ddc411b60264","Type":"ContainerDied","Data":"c22ebae849802bf78172cd881d17d58003f4269f0d080a8f2dd1ee3b5633d3b7"}
Jul 28 12:54:17.677502 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.677483    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" event={"ID":"51477dbb-d4b7-448e-9583-64f2d909364d","Type":"ContainerStarted","Data":"274d08a47dda1091580d7d24318c0b7b885b7fa82795b7edd5d54681fa495888"}
Jul 28 12:54:17.677765 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.677742    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:54:17.677887 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.677809    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:54:17.677887 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.677856    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:54:17.692503 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.692482    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:54:17.692612 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.692568    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:54:17.729155 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:17.729077    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh" podStartSLOduration=9.267637149 podStartE2EDuration="27.729063305s" totalImagesPullingTime="18.461426156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:17.728567666 +0000 UTC m=+27.765936572" watchObservedRunningTime="2026-07-28 12:54:17.729063305 +0000 UTC m=+27.766432241"
Jul 28 12:54:18.470666 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:18.470636    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:18.470854 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:18.470739    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:19.471300 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:19.471268    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:19.471300 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:19.471304    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:19.471686 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:19.471387    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:19.471686 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:19.471461    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:19.670492 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:19.670467    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:19.824382 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:19.822688    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:19.941689 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:19.941654    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:20.042691 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:20.042608    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:20.167861 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:20.167833    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:20.253349 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:20.253320    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:20.346696 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:20.346617    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:20.471855 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:20.471828    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:20.472206 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:20.471914    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:20.686544 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:20.686516    2553 generic.go:356] "Generic (PLEG): container finished" podID="14d239a1-4dfe-4102-b8e5-ddc411b60264" containerID="2a2caae9e894ce68b171c2c5c2f493db677a5e7b4c68b3d72ecf7e1cb31170cc" exitCode=0
Jul 28 12:54:20.686672 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:20.686565    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" event={"ID":"14d239a1-4dfe-4102-b8e5-ddc411b60264","Type":"ContainerDied","Data":"2a2caae9e894ce68b171c2c5c2f493db677a5e7b4c68b3d72ecf7e1cb31170cc"}
Jul 28 12:54:21.096274 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:21.096202    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:21.470731 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:21.470700    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:21.470891 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:21.470704    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:21.470891 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:21.470841    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:21.470962 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:21.470885    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:21.688076 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:21.688040    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:21.891118 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:21.891088    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:22.107482 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:22.107397    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:22.470958 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:22.470931    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:22.471100 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:22.471070    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:22.501867 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:22.501834    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:22.692612 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:22.692584    2553 generic.go:356] "Generic (PLEG): container finished" podID="14d239a1-4dfe-4102-b8e5-ddc411b60264" containerID="0317bb5b492424725e8bc7387187c01e9edd655d107a55b9c7e44164d8c656f7" exitCode=0
Jul 28 12:54:22.692940 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:22.692622    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" event={"ID":"14d239a1-4dfe-4102-b8e5-ddc411b60264","Type":"ContainerDied","Data":"0317bb5b492424725e8bc7387187c01e9edd655d107a55b9c7e44164d8c656f7"}
Jul 28 12:54:23.096720 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:23.096686    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:23.299066 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:23.299018    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:23.470745 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:23.470713    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:23.470941 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:23.470759    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:23.470941 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:23.470845    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:23.471023 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:23.470977    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:23.503006 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:23.502959    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:23.563451 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:23.563415    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:23.894437 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:23.894338    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:24.056846 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:24.056810    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:24.057005 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:24.056953    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:54:24.057050 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:24.057018    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret podName:2058fd10-4ec4-45cd-b41f-3496159fa93c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:56.056997141 +0000 UTC m=+66.094366026 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret") pod "global-pull-secret-syncer-2f4qz" (UID: "2058fd10-4ec4-45cd-b41f-3496159fa93c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:54:24.119950 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:24.119903    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:24.158013 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:24.157922    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:24.158149 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:24.158106    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:54:24.158203 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:24.158183    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs podName:1bece3c2-4eb4-4219-a757-45a57c351842 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:56.158163768 +0000 UTC m=+66.195532668 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs") pod "network-metrics-daemon-qrq22" (UID: "1bece3c2-4eb4-4219-a757-45a57c351842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:54:24.223117 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:24.223066    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:24.258888 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:24.258847    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:24.259045 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:24.259029    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:54:24.259101 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:24.259058    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:54:24.259101 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:24.259071    2553 projected.go:196] Error preparing data for projected volume kube-api-access-zbld7 for pod openshift-network-diagnostics/network-check-target-cqxxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:54:24.259199 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:24.259136    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7 podName:2ebe21a0-6999-409b-a63b-0fd0265d25b5 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:56.259118427 +0000 UTC m=+66.296487332 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zbld7" (UniqueName: "kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7") pod "network-check-target-cqxxn" (UID: "2ebe21a0-6999-409b-a63b-0fd0265d25b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:54:24.471160 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:24.471116    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:24.471319 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:24.471243    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:24.495226 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:24.495187    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:25.096785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:25.096742    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:25.471040 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:25.471009    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:25.471205 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:25.471052    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:25.471205 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:25.471141    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:25.471325 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:25.471266    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:25.496962 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:25.496929    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:25.550624 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:25.550564    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:25.691978 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:25.691783    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:25.901233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:25.901142    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:26.470936 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:26.470905    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:26.471342 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:26.471007    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:26.704892 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:26.704860    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" event={"ID":"14d239a1-4dfe-4102-b8e5-ddc411b60264","Type":"ContainerStarted","Data":"6d0cbc03efbd9dc82612148d83d6bc26c8dd575da3824bcc3eb46dc4a0cf31d6"}
Jul 28 12:54:26.894881 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:26.894784    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:27.294109 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:27.294078    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:27.395900 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:27.395865    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:27.471306 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:27.471281    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:27.471745 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:27.471402    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:27.471745 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:27.471449    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:27.471745 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:27.471533    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:27.495033 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:27.495003    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:27.643183 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:27.643078    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:27.709888 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:27.709859    2553 generic.go:356] "Generic (PLEG): container finished" podID="14d239a1-4dfe-4102-b8e5-ddc411b60264" containerID="6d0cbc03efbd9dc82612148d83d6bc26c8dd575da3824bcc3eb46dc4a0cf31d6" exitCode=0
Jul 28 12:54:27.710068 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:27.709907    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" event={"ID":"14d239a1-4dfe-4102-b8e5-ddc411b60264","Type":"ContainerDied","Data":"6d0cbc03efbd9dc82612148d83d6bc26c8dd575da3824bcc3eb46dc4a0cf31d6"}
Jul 28 12:54:27.738758 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:27.738727    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:28.108810 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:28.108766    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:28.470333 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:28.470301    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:28.470480 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:28.470407    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:28.694107 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:28.694070    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:28.715200 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:28.715159    2553 generic.go:356] "Generic (PLEG): container finished" podID="14d239a1-4dfe-4102-b8e5-ddc411b60264" containerID="d2ca4e3c4016e549085714cd93e3e913914ec193d9ceaa93526491a2e3718664" exitCode=0
Jul 28 12:54:28.715344 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:28.715211    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" event={"ID":"14d239a1-4dfe-4102-b8e5-ddc411b60264","Type":"ContainerDied","Data":"d2ca4e3c4016e549085714cd93e3e913914ec193d9ceaa93526491a2e3718664"}
Jul 28 12:54:29.091859 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:29.091763    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:29.289581 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:29.289542    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:29.470604 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:29.470578    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:29.470749 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:29.470695    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:29.470831 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:29.470743    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:29.470878 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:29.470834    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:29.720243 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:29.720208    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" event={"ID":"14d239a1-4dfe-4102-b8e5-ddc411b60264","Type":"ContainerStarted","Data":"813a82850ae43dcb8b647450d1e35f3cf9913d959d442f49db55fc12e3b1b0a1"}
Jul 28 12:54:29.744679 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:29.744599    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5wdsp" podStartSLOduration=10.153187364 podStartE2EDuration="39.744583202s" totalImagesPullingTime="29.591395838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:29.743206224 +0000 UTC m=+39.780575164" watchObservedRunningTime="2026-07-28 12:54:29.744583202 +0000 UTC m=+39.781952109"
Jul 28 12:54:29.892168 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:29.892137    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:30.100279 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:30.100177    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:30.296101 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:30.296061    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:30.471814 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:30.471767    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:30.471996 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:30.471884    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:31.470952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:31.470917    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:31.471403 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:31.470920    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:31.471403 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:31.471012    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:31.471403 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:31.471146    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:32.470693 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:32.470666    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:32.470855 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:32.470767    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:33.198584 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:33.198552    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:33.280720 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:33.280689    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:33.471237 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:33.471163    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:33.471369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:33.471166    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:33.471369 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:33.471257    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:33.471369 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:33.471334    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:34.471220 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:34.471193    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:34.471595 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:34.471286    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:35.471017 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:35.470981    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:35.471278 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:35.471033    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:35.471278 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:35.471104    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:35.471278 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:35.471240    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:36.470911 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:36.470874    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:36.471090 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:36.470976    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:37.470587 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:37.470552    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:37.470989 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:37.470657    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:37.470989 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:37.470712    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:37.470989 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:37.470785    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:37.806691 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:37.806600    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:37.900608 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:37.900573    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:37.999185 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:37.999151    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:38.103102 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:38.102996    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:38.199922 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:38.199862    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:38.470672 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:38.470638    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:38.471098 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:38.470772    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:38.965734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:38.965708    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:39.113885 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:39.113856    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cqxxn"]
Jul 28 12:54:39.114669 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:39.114252    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:39.114669 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:39.114363    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:39.114669 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:39.114417    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qrq22"]
Jul 28 12:54:39.114669 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:39.114515    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:39.114669 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:39.114619    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:39.115188 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:39.115122    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2f4qz"]
Jul 28 12:54:39.115279 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:39.115206    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:39.115343 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:39.115291    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:39.126574 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:39.126495    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:39.491333 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:39.491292    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:39.929235 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:39.929201    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:40.032313 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:40.032277    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:40.234269 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:40.234192    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:40.473421 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:40.473394    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:40.473570 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:40.473481    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:40.473570 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:40.473508    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4qz" podUID="2058fd10-4ec4-45cd-b41f-3496159fa93c"
Jul 28 12:54:40.473647 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:40.473573    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qrq22" podUID="1bece3c2-4eb4-4219-a757-45a57c351842"
Jul 28 12:54:41.264169 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:41.264136    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:41.324666 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:41.324629    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:41.471064 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:41.471037    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:41.471263 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:41.471154    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqxxn" podUID="2ebe21a0-6999-409b-a63b-0fd0265d25b5"
Jul 28 12:54:41.504289 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:41.504259    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:41.609107 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:41.609019    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:42.274518 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.274441    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-149.ec2.internal" event="NodeReady"
Jul 28 12:54:42.274975 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.274559    2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:54:42.307808 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.307763    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:42.325412 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.325385    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-td7h2"]
Jul 28 12:54:42.340519 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.340493    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mzdrz"]
Jul 28 12:54:42.340616 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.340597    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.349110 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.349093    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:54:42.349219 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.349112    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-blfqk\""
Jul 28 12:54:42.349219 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.349149    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:54:42.349219 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.349186    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:54:42.349318 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.349227    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:54:42.354349 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.354333    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzdrz"
Jul 28 12:54:42.356829 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.356789    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-td7h2"]
Jul 28 12:54:42.357458 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.357437    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzdrz"]
Jul 28 12:54:42.359596 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.359575    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:54:42.359816 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.359778    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:54:42.359901 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.359818    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gt9sx\""
Jul 28 12:54:42.360303 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.360282    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:54:42.394501 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.394476    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59199dc1-6c80-47c3-a6a4-fb9bcef71268-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.394501 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.394503    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwbd\" (UniqueName: \"kubernetes.io/projected/59199dc1-6c80-47c3-a6a4-fb9bcef71268-kube-api-access-dpwbd\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.394648 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.394524    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/111434a7-0841-42b1-85b1-cd16ab0eada9-cert\") pod \"ingress-canary-mzdrz\" (UID: \"111434a7-0841-42b1-85b1-cd16ab0eada9\") " pod="openshift-ingress-canary/ingress-canary-mzdrz"
Jul 28 12:54:42.394648 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.394540    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hv2\" (UniqueName: \"kubernetes.io/projected/111434a7-0841-42b1-85b1-cd16ab0eada9-kube-api-access-m8hv2\") pod \"ingress-canary-mzdrz\" (UID: \"111434a7-0841-42b1-85b1-cd16ab0eada9\") " pod="openshift-ingress-canary/ingress-canary-mzdrz"
Jul 28 12:54:42.394648 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.394622    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59199dc1-6c80-47c3-a6a4-fb9bcef71268-crio-socket\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.394648 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.394642    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59199dc1-6c80-47c3-a6a4-fb9bcef71268-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.394782 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.394676    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59199dc1-6c80-47c3-a6a4-fb9bcef71268-data-volume\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.436397 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.436375    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2hlrq"]
Jul 28 12:54:42.449098 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.449072    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:42.452494 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.452480    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.461313 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.461293    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:54:42.461445 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.461301    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xjd7z\""
Jul 28 12:54:42.461694 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.461679    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:54:42.462086 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.462071    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hlrq"]
Jul 28 12:54:42.470734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.470715    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:42.470875 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.470860    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:42.473969 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.473951    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z7m8f\""
Jul 28 12:54:42.474089 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.474018    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:54:42.474371 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.474355    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:54:42.494935 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.494913    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1725782f-575b-4eea-bca2-58b8de98444d-metrics-tls\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.495022 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.494951    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59199dc1-6c80-47c3-a6a4-fb9bcef71268-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.495022 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.494969    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwbd\" (UniqueName: \"kubernetes.io/projected/59199dc1-6c80-47c3-a6a4-fb9bcef71268-kube-api-access-dpwbd\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.495022 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495015    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/111434a7-0841-42b1-85b1-cd16ab0eada9-cert\") pod \"ingress-canary-mzdrz\" (UID: \"111434a7-0841-42b1-85b1-cd16ab0eada9\") " pod="openshift-ingress-canary/ingress-canary-mzdrz"
Jul 28 12:54:42.495133 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495043    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hv2\" (UniqueName: \"kubernetes.io/projected/111434a7-0841-42b1-85b1-cd16ab0eada9-kube-api-access-m8hv2\") pod \"ingress-canary-mzdrz\" (UID: \"111434a7-0841-42b1-85b1-cd16ab0eada9\") " pod="openshift-ingress-canary/ingress-canary-mzdrz"
Jul 28 12:54:42.495133 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495072    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1725782f-575b-4eea-bca2-58b8de98444d-tmp-dir\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.495220 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495151    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59199dc1-6c80-47c3-a6a4-fb9bcef71268-crio-socket\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.495220 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495176    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59199dc1-6c80-47c3-a6a4-fb9bcef71268-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.495301 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495231    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxfb\" (UniqueName: \"kubernetes.io/projected/1725782f-575b-4eea-bca2-58b8de98444d-kube-api-access-jvxfb\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.495348 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495298    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59199dc1-6c80-47c3-a6a4-fb9bcef71268-data-volume\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.495348 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495337    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1725782f-575b-4eea-bca2-58b8de98444d-config-volume\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.495421 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495347    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59199dc1-6c80-47c3-a6a4-fb9bcef71268-crio-socket\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.495570 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495551    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59199dc1-6c80-47c3-a6a4-fb9bcef71268-data-volume\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.495690 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.495661    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59199dc1-6c80-47c3-a6a4-fb9bcef71268-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.498723 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.498705    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59199dc1-6c80-47c3-a6a4-fb9bcef71268-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.498858 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.498839    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/111434a7-0841-42b1-85b1-cd16ab0eada9-cert\") pod \"ingress-canary-mzdrz\" (UID: \"111434a7-0841-42b1-85b1-cd16ab0eada9\") " pod="openshift-ingress-canary/ingress-canary-mzdrz"
Jul 28 12:54:42.504620 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.504584    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hv2\" (UniqueName: \"kubernetes.io/projected/111434a7-0841-42b1-85b1-cd16ab0eada9-kube-api-access-m8hv2\") pod \"ingress-canary-mzdrz\" (UID: \"111434a7-0841-42b1-85b1-cd16ab0eada9\") " pod="openshift-ingress-canary/ingress-canary-mzdrz"
Jul 28 12:54:42.504826 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.504786    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwbd\" (UniqueName: \"kubernetes.io/projected/59199dc1-6c80-47c3-a6a4-fb9bcef71268-kube-api-access-dpwbd\") pod \"insights-runtime-extractor-td7h2\" (UID: \"59199dc1-6c80-47c3-a6a4-fb9bcef71268\") " pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.596572 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.596499    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1725782f-575b-4eea-bca2-58b8de98444d-tmp-dir\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.596714 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.596581    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxfb\" (UniqueName: \"kubernetes.io/projected/1725782f-575b-4eea-bca2-58b8de98444d-kube-api-access-jvxfb\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.596714 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.596611    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1725782f-575b-4eea-bca2-58b8de98444d-config-volume\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.596714 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.596632    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1725782f-575b-4eea-bca2-58b8de98444d-metrics-tls\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.596885 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.596867    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1725782f-575b-4eea-bca2-58b8de98444d-tmp-dir\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.597158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.597125    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1725782f-575b-4eea-bca2-58b8de98444d-config-volume\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.598832 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.598788    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1725782f-575b-4eea-bca2-58b8de98444d-metrics-tls\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.606430 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.606404    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxfb\" (UniqueName: \"kubernetes.io/projected/1725782f-575b-4eea-bca2-58b8de98444d-kube-api-access-jvxfb\") pod \"dns-default-2hlrq\" (UID: \"1725782f-575b-4eea-bca2-58b8de98444d\") " pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.650130 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.650098    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-td7h2"
Jul 28 12:54:42.663216 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.663191    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzdrz"
Jul 28 12:54:42.761035 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.761004    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:42.836965 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.836932    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzdrz"]
Jul 28 12:54:42.844533 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.844504    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-td7h2"]
Jul 28 12:54:42.846558 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:42.846536    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59199dc1_6c80_47c3_a6a4_fb9bcef71268.slice/crio-3e70c5dbc367e756f46eb4b309a4509aece2ccbec8681dfc2902ddf6174eb411 WatchSource:0}: Error finding container 3e70c5dbc367e756f46eb4b309a4509aece2ccbec8681dfc2902ddf6174eb411: Status 404 returned error can't find the container with id 3e70c5dbc367e756f46eb4b309a4509aece2ccbec8681dfc2902ddf6174eb411
Jul 28 12:54:42.897506 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:42.897466    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hlrq"]
Jul 28 12:54:42.901198 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:42.901173    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1725782f_575b_4eea_bca2_58b8de98444d.slice/crio-34e77b5a8cba6796a54b5f90014463334920a4ec339db683ea075aa5406231bd WatchSource:0}: Error finding container 34e77b5a8cba6796a54b5f90014463334920a4ec339db683ea075aa5406231bd: Status 404 returned error can't find the container with id 34e77b5a8cba6796a54b5f90014463334920a4ec339db683ea075aa5406231bd
Jul 28 12:54:43.217245 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.217210    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-btb6q"]
Jul 28 12:54:43.228059 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.228039    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.233311 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.233291    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vcnl9\""
Jul 28 12:54:43.233884 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.233863    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 12:54:43.234486 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.234461    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:54:43.234723 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.234708    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 12:54:43.234963 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.234942    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:54:43.237555 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.237538    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:54:43.238438 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.238419    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-btb6q"]
Jul 28 12:54:43.304595 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.304560    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.305094 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.304691    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkdn\" (UniqueName: \"kubernetes.io/projected/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-kube-api-access-qmkdn\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.305094 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.304734    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.305094 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.304766    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.405232 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.405190    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkdn\" (UniqueName: \"kubernetes.io/projected/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-kube-api-access-qmkdn\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.405401 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.405252    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.405401 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.405293    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.405401 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.405330    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.406716 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.406666    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.410679 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.410655    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.411242 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.411216    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.417369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.417343    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkdn\" (UniqueName: \"kubernetes.io/projected/b3b3e3df-3308-4e68-b285-d82cbce5fd3e-kube-api-access-qmkdn\") pod \"prometheus-operator-84d69f794d-btb6q\" (UID: \"b3b3e3df-3308-4e68-b285-d82cbce5fd3e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.471249 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.471176    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:43.475597 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.475569    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kmztt\""
Jul 28 12:54:43.475907 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.475878    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:54:43.476093 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.476077    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:54:43.481239 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.481207    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:43.537322 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.537292    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q"
Jul 28 12:54:43.546750 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.546718    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:43.727754 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.727680    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-btb6q"]
Jul 28 12:54:43.736930 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:43.736889    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b3e3df_3308_4e68_b285_d82cbce5fd3e.slice/crio-eceb4dbbfb1f5ff46ffa2cabf7da6ef915eedb00c3faf6b2861b00a88990f9d5 WatchSource:0}: Error finding container eceb4dbbfb1f5ff46ffa2cabf7da6ef915eedb00c3faf6b2861b00a88990f9d5: Status 404 returned error can't find the container with id eceb4dbbfb1f5ff46ffa2cabf7da6ef915eedb00c3faf6b2861b00a88990f9d5
Jul 28 12:54:43.756301 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.756269    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q" event={"ID":"b3b3e3df-3308-4e68-b285-d82cbce5fd3e","Type":"ContainerStarted","Data":"eceb4dbbfb1f5ff46ffa2cabf7da6ef915eedb00c3faf6b2861b00a88990f9d5"}
Jul 28 12:54:43.758013 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.757988    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hlrq" event={"ID":"1725782f-575b-4eea-bca2-58b8de98444d","Type":"ContainerStarted","Data":"34e77b5a8cba6796a54b5f90014463334920a4ec339db683ea075aa5406231bd"}
Jul 28 12:54:43.759395 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.759367    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-td7h2" event={"ID":"59199dc1-6c80-47c3-a6a4-fb9bcef71268","Type":"ContainerStarted","Data":"3e70c5dbc367e756f46eb4b309a4509aece2ccbec8681dfc2902ddf6174eb411"}
Jul 28 12:54:43.761209 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:43.761186    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzdrz" event={"ID":"111434a7-0841-42b1-85b1-cd16ab0eada9","Type":"ContainerStarted","Data":"b4f42c36bdc96598970b30c640d1b6df617a31e5cc68367090b08a38e5490ec9"}
Jul 28 12:54:44.956035 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:44.955882    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:45.019238 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:45.019184    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:49.185189 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.185150    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:49.286542 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.286513    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:49.353968 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.353936    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:49.698706 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.698677    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hmhh"
Jul 28 12:54:49.722056 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.722023    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:49.781061 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.781022    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzdrz" event={"ID":"111434a7-0841-42b1-85b1-cd16ab0eada9","Type":"ContainerStarted","Data":"580d8ff62a6ad01cfda38b652884c23ef9363e6b2f7ade61b8fbfc0b0b66966d"}
Jul 28 12:54:49.783435 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.783397    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q" event={"ID":"b3b3e3df-3308-4e68-b285-d82cbce5fd3e","Type":"ContainerStarted","Data":"7250ed977a4e33ff6a594788f1b9595cbba5ec126071fddd0772e970632fc3a4"}
Jul 28 12:54:49.783555 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.783450    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q" event={"ID":"b3b3e3df-3308-4e68-b285-d82cbce5fd3e","Type":"ContainerStarted","Data":"ff51210dff55e435a941ec9283508fc74071483363a24ae0758f401facb51160"}
Jul 28 12:54:49.785558 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.785533    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hlrq" event={"ID":"1725782f-575b-4eea-bca2-58b8de98444d","Type":"ContainerStarted","Data":"c24555d4effdcc81a3d5e954948e7a6086ed2da8603251abbceefc05fdd4d264"}
Jul 28 12:54:49.785681 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.785578    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hlrq" event={"ID":"1725782f-575b-4eea-bca2-58b8de98444d","Type":"ContainerStarted","Data":"67e98b238360b59c471d6a131ac27ee909d30e02efe060767a3bb07d0774700d"}
Jul 28 12:54:49.785681 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.785609    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:54:49.787543 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.787509    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-td7h2" event={"ID":"59199dc1-6c80-47c3-a6a4-fb9bcef71268","Type":"ContainerStarted","Data":"9c4cd63bc6078871db5a17bdbe733e7c03668ce6e43d5f45b3f85e4656b0e6ca"}
Jul 28 12:54:49.801459 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.801413    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mzdrz" podStartSLOduration=1.844098609 podStartE2EDuration="7.801394999s" totalImagesPullingTime="5.95729639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:49.800149804 +0000 UTC m=+59.837518708" watchObservedRunningTime="2026-07-28 12:54:49.801394999 +0000 UTC m=+59.838763919"
Jul 28 12:54:49.809293 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.809121    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:49.839814 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.839752    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2hlrq" podStartSLOduration=1.943688528 podStartE2EDuration="7.839735529s" totalImagesPullingTime="5.896047001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:49.839586993 +0000 UTC m=+59.876955892" watchObservedRunningTime="2026-07-28 12:54:49.839735529 +0000 UTC m=+59.877104434"
Jul 28 12:54:49.839956 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.839909    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-btb6q" podStartSLOduration=1.7781943180000002 podStartE2EDuration="6.839900506s" totalImagesPullingTime="5.061706188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:49.819206375 +0000 UTC m=+59.856575283" watchObservedRunningTime="2026-07-28 12:54:49.839900506 +0000 UTC m=+59.877269414"
Jul 28 12:54:49.882758 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.882605    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:49.945933 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:49.945897    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:51.377594 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.377555    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:51.536148 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.536120    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:51.572482 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.572447    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"]
Jul 28 12:54:51.574325 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.574303    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.574656 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.574639    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"]
Jul 28 12:54:51.577741 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.577714    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.579921 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.579894    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 12:54:51.579921 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.579900    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hbkl9\""
Jul 28 12:54:51.580078 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.580041    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 12:54:51.580831 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.580809    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 12:54:51.581311 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.581290    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-swmwp\""
Jul 28 12:54:51.581637 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.581616    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 12:54:51.581717 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.581646    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 12:54:51.591007 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.590986    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"]
Jul 28 12:54:51.594452 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.594430    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"]
Jul 28 12:54:51.628823 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.628742    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6drlp"]
Jul 28 12:54:51.631521 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.631496    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.634374 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.634351    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:54:51.634489 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.634435    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:54:51.634738 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.634719    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bjvgr\""
Jul 28 12:54:51.634995 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.634970    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:54:51.657652 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.657623    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:51.681672 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681629    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a710ec85-fda1-47b0-a06c-6d5539f709c8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.681838 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681749    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.681838 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681775    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5ch\" (UniqueName: \"kubernetes.io/projected/a710ec85-fda1-47b0-a06c-6d5539f709c8-kube-api-access-rr5ch\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.681838 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681821    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a73e5a1-f9e8-4c93-9790-b1315e823e13-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.682004 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681846    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a73e5a1-f9e8-4c93-9790-b1315e823e13-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.682004 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681872    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.682004 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681895    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrlr5\" (UniqueName: \"kubernetes.io/projected/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-api-access-xrlr5\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.682004 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681928    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a710ec85-fda1-47b0-a06c-6d5539f709c8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.682004 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681944    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a710ec85-fda1-47b0-a06c-6d5539f709c8-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.682004 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.681964    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.752037 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.751999    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:51.783165 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783132    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spn65\" (UniqueName: \"kubernetes.io/projected/addd1058-8188-4cb5-b23a-e25a8dfb393c-kube-api-access-spn65\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.783320 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783174    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a73e5a1-f9e8-4c93-9790-b1315e823e13-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.783320 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783203    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-textfile\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.783320 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/addd1058-8188-4cb5-b23a-e25a8dfb393c-root\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.783320 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783275    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a710ec85-fda1-47b0-a06c-6d5539f709c8-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.783476 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783327    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a710ec85-fda1-47b0-a06c-6d5539f709c8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.783476 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783376    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.783476 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783422    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.783476 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783447    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-tls\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.783476 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783472    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5ch\" (UniqueName: \"kubernetes.io/projected/a710ec85-fda1-47b0-a06c-6d5539f709c8-kube-api-access-rr5ch\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.783677 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783502    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a73e5a1-f9e8-4c93-9790-b1315e823e13-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.783677 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783529    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrlr5\" (UniqueName: \"kubernetes.io/projected/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-api-access-xrlr5\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.783677 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783553    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-accelerators-collector-config\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.783677 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783588    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.783677 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783627    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/addd1058-8188-4cb5-b23a-e25a8dfb393c-sys\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.783677 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783653    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.783913 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783677    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-wtmp\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.783913 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783699    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/addd1058-8188-4cb5-b23a-e25a8dfb393c-metrics-client-ca\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.783913 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783746    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a710ec85-fda1-47b0-a06c-6d5539f709c8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.784044 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.783956    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a73e5a1-f9e8-4c93-9790-b1315e823e13-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.784044 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:51.784003    2553 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 12:54:51.784138 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:51.784092    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a710ec85-fda1-47b0-a06c-6d5539f709c8-openshift-state-metrics-tls podName:a710ec85-fda1-47b0-a06c-6d5539f709c8 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:52.284071561 +0000 UTC m=+62.321440451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a710ec85-fda1-47b0-a06c-6d5539f709c8-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-srvzp" (UID: "a710ec85-fda1-47b0-a06c-6d5539f709c8") : secret "openshift-state-metrics-tls" not found
Jul 28 12:54:51.784197 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.784138    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6a73e5a1-f9e8-4c93-9790-b1315e823e13-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.784402 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.784378    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.784761 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.784736    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a710ec85-fda1-47b0-a06c-6d5539f709c8-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.787328 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.787302    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.787428 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.787393    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a710ec85-fda1-47b0-a06c-6d5539f709c8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.787428 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.787423    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.794611 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.794593    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrlr5\" (UniqueName: \"kubernetes.io/projected/6a73e5a1-f9e8-4c93-9790-b1315e823e13-kube-api-access-xrlr5\") pod \"kube-state-metrics-56d9f944b4-kp9k5\" (UID: \"6a73e5a1-f9e8-4c93-9790-b1315e823e13\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.794973 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.794951    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5ch\" (UniqueName: \"kubernetes.io/projected/a710ec85-fda1-47b0-a06c-6d5539f709c8-kube-api-access-rr5ch\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:51.885017 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.884933    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spn65\" (UniqueName: \"kubernetes.io/projected/addd1058-8188-4cb5-b23a-e25a8dfb393c-kube-api-access-spn65\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885017 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.884990    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-textfile\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885231 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885017    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/addd1058-8188-4cb5-b23a-e25a8dfb393c-root\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885231 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885066    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885231 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885119    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-tls\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885231 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885162    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-accelerators-collector-config\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885231 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885172    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/addd1058-8188-4cb5-b23a-e25a8dfb393c-root\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885231 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885198    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/addd1058-8188-4cb5-b23a-e25a8dfb393c-sys\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885231 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885233    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/addd1058-8188-4cb5-b23a-e25a8dfb393c-sys\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885550 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885243    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-wtmp\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885550 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885272    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/addd1058-8188-4cb5-b23a-e25a8dfb393c-metrics-client-ca\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885550 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885304    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-textfile\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885550 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:51.885388    2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 12:54:51.885550 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885414    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-wtmp\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885550 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:54:51.885432    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-tls podName:addd1058-8188-4cb5-b23a-e25a8dfb393c nodeName:}" failed. No retries permitted until 2026-07-28 12:54:52.385418262 +0000 UTC m=+62.422787147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-tls") pod "node-exporter-6drlp" (UID: "addd1058-8188-4cb5-b23a-e25a8dfb393c") : secret "node-exporter-tls" not found
Jul 28 12:54:51.885850 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885817    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/addd1058-8188-4cb5-b23a-e25a8dfb393c-metrics-client-ca\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.885978 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.885957    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-accelerators-collector-config\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.888360 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.888336    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.895353 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.895329    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"
Jul 28 12:54:51.904306 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.904284    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spn65\" (UniqueName: \"kubernetes.io/projected/addd1058-8188-4cb5-b23a-e25a8dfb393c-kube-api-access-spn65\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:51.942572 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:51.942538    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:52.045766 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.045730    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:52.139059 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.138982    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:52.289443 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.289416    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a710ec85-fda1-47b0-a06c-6d5539f709c8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:52.292102 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.292057    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a710ec85-fda1-47b0-a06c-6d5539f709c8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-srvzp\" (UID: \"a710ec85-fda1-47b0-a06c-6d5539f709c8\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:52.365297 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.365213    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5"]
Jul 28 12:54:52.367552 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:52.367526    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a73e5a1_f9e8_4c93_9790_b1315e823e13.slice/crio-d1d74b23d07277e3cb2560f583938887999eb82530cc04b25e69947559c236ac WatchSource:0}: Error finding container d1d74b23d07277e3cb2560f583938887999eb82530cc04b25e69947559c236ac: Status 404 returned error can't find the container with id d1d74b23d07277e3cb2560f583938887999eb82530cc04b25e69947559c236ac
Jul 28 12:54:52.383629 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.383606    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:52.390158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.390110    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-tls\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:52.392118 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.392094    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/addd1058-8188-4cb5-b23a-e25a8dfb393c-node-exporter-tls\") pod \"node-exporter-6drlp\" (UID: \"addd1058-8188-4cb5-b23a-e25a8dfb393c\") " pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:52.488365 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.488340    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"
Jul 28 12:54:52.540677 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.540530    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6drlp"
Jul 28 12:54:52.574728 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.574692    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:52.624281 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.624252    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp"]
Jul 28 12:54:52.627372 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:52.627346    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda710ec85_fda1_47b0_a06c_6d5539f709c8.slice/crio-bb9054dba091a24ed3ff2c2d6a4cc66cd84e55b192c1c805d4afa822bac009f5 WatchSource:0}: Error finding container bb9054dba091a24ed3ff2c2d6a4cc66cd84e55b192c1c805d4afa822bac009f5: Status 404 returned error can't find the container with id bb9054dba091a24ed3ff2c2d6a4cc66cd84e55b192c1c805d4afa822bac009f5
Jul 28 12:54:52.665916 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.665883    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:52.676568 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.676540    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:54:52.680812 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.680774    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.684909 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.684564    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 12:54:52.684909 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.684697    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 12:54:52.684909 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.684703    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 12:54:52.684909 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.684703    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 12:54:52.684909 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.684964    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 12:54:52.685376 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.685330    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 12:54:52.685431 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.685406    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ptmgh\""
Jul 28 12:54:52.685482 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.685330    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 12:54:52.685582 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.685463    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 12:54:52.685897 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.685881    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 12:54:52.703788 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.703234    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:54:52.792903 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.792862    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793059 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.792916    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-config-out\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793059 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.792970    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793174 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793109    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793174 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793163    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793272 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793191    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793272 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793222    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-web-config\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793272 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793244    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km72t\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-kube-api-access-km72t\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793418 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793294    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793418 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793333    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793418 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793366    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793418 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793390    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.793686 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.793418    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.800460 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.800425    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6drlp" event={"ID":"addd1058-8188-4cb5-b23a-e25a8dfb393c","Type":"ContainerStarted","Data":"fda11bb6ad1d70ccc4a95c30a6b6805844b4820bb6702de604b47a7efe76a303"}
Jul 28 12:54:52.802768 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.802737    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-td7h2" event={"ID":"59199dc1-6c80-47c3-a6a4-fb9bcef71268","Type":"ContainerStarted","Data":"0c1cd8bcaafd2c9197d9b3505ad3bf5dd4a33eea921f696a367b76a0ff7a9d45"}
Jul 28 12:54:52.806104 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.806053    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp" event={"ID":"a710ec85-fda1-47b0-a06c-6d5539f709c8","Type":"ContainerStarted","Data":"bf1529cafdc1426f9fd2956c75f75eb20cf05111663630dac10ced389c53d08a"}
Jul 28 12:54:52.806104 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.806098    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp" event={"ID":"a710ec85-fda1-47b0-a06c-6d5539f709c8","Type":"ContainerStarted","Data":"6bb661f1c3cf6401efdd297cf32b496abeb4adec8f34fcc3231b369ce477cf19"}
Jul 28 12:54:52.806297 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.806114    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp" event={"ID":"a710ec85-fda1-47b0-a06c-6d5539f709c8","Type":"ContainerStarted","Data":"bb9054dba091a24ed3ff2c2d6a4cc66cd84e55b192c1c805d4afa822bac009f5"}
Jul 28 12:54:52.808029 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.808004    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5" event={"ID":"6a73e5a1-f9e8-4c93-9790-b1315e823e13","Type":"ContainerStarted","Data":"d1d74b23d07277e3cb2560f583938887999eb82530cc04b25e69947559c236ac"}
Jul 28 12:54:52.896509 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896412    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896509 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896473    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896509 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896500    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896531    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-web-config\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896556    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km72t\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-kube-api-access-km72t\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896602    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896640    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896670    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896694    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896724    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896755    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.896785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896789    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-config-out\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.897266 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.896835    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.897266 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.897185    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.898001 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.897653    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.899008 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.898979    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.900202 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.900168    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.901272 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.901225    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-web-config\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.901272 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.901265    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.901967 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.901940    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.902341 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.902293    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.902439 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.902386    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-config-out\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.902630 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.902609    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.902811 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.902769    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.903048 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.903033    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.908291 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.908269    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km72t\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-kube-api-access-km72t\") pod \"alertmanager-main-0\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:52.992480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:52.992440    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:54:53.163660 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.163558    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:54:53.167301 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:53.167268    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f930747_3471_4519_afeb_6620221b95a9.slice/crio-4e861a47f909bcd12f82c6aeb67f400a68760437b51309372e479539c1ae45cc WatchSource:0}: Error finding container 4e861a47f909bcd12f82c6aeb67f400a68760437b51309372e479539c1ae45cc: Status 404 returned error can't find the container with id 4e861a47f909bcd12f82c6aeb67f400a68760437b51309372e479539c1ae45cc
Jul 28 12:54:53.680767 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.680728    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"]
Jul 28 12:54:53.684640 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.684543    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.688301 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.688222    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 12:54:53.688895 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.688614    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 12:54:53.688895 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.688614    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 12:54:53.688895 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.688658    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 12:54:53.688895 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.688828    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tr2hz\""
Jul 28 12:54:53.689179 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.689165    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 12:54:53.689711 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.689690    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ei8jdoonhbivc\""
Jul 28 12:54:53.695891 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.695787    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"]
Jul 28 12:54:53.773787 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.773745    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:53.804829 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.804776    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdw44\" (UniqueName: \"kubernetes.io/projected/19fad7fd-e532-475d-b891-db67a414256c-kube-api-access-xdw44\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.805011 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.804880    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-grpc-tls\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.805011 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.804925    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19fad7fd-e532-475d-b891-db67a414256c-metrics-client-ca\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.805011 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.804959    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.805193 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.805096    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.805193 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.805121    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-tls\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.805193 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.805153    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.805334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.805220    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.814808 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.814748    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerStarted","Data":"4e861a47f909bcd12f82c6aeb67f400a68760437b51309372e479539c1ae45cc"}
Jul 28 12:54:53.874317 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.874267    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:53.906379 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.906343    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdw44\" (UniqueName: \"kubernetes.io/projected/19fad7fd-e532-475d-b891-db67a414256c-kube-api-access-xdw44\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.906536 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.906399    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-grpc-tls\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.906536 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.906443    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19fad7fd-e532-475d-b891-db67a414256c-metrics-client-ca\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.906536 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.906476    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.906696 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.906561    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.906765 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.906731    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-tls\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.906836 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.906777    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.907038 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.906858    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.907347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.907316    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19fad7fd-e532-475d-b891-db67a414256c-metrics-client-ca\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.909474 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.909428    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.909882 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.909852    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.910087 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.910071    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.910150 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.910080    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.910150 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.910100    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-thanos-querier-tls\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.910319 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.910289    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/19fad7fd-e532-475d-b891-db67a414256c-secret-grpc-tls\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.915167 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.915144    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdw44\" (UniqueName: \"kubernetes.io/projected/19fad7fd-e532-475d-b891-db67a414256c-kube-api-access-xdw44\") pod \"thanos-querier-7c8fdfcf97-7h2kz\" (UID: \"19fad7fd-e532-475d-b891-db67a414256c\") " pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:53.997556 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:53.997462    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:54:54.378571 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:54.378530    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"]
Jul 28 12:54:54.579610 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:54.579515    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:54.644497 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:54.644467    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:54.712788 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:54.712756    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:54.820595 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:54.820552    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp" event={"ID":"a710ec85-fda1-47b0-a06c-6d5539f709c8","Type":"ContainerStarted","Data":"bc35731995935ff0dea5a3354c5f7e925a7d516c045eb309aa7c6e91b0da8d86"}
Jul 28 12:54:54.840629 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:54.840522    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-srvzp" podStartSLOduration=2.387357853 podStartE2EDuration="3.840502232s" totalImagesPullingTime="1.453144379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:54.83931745 +0000 UTC m=+64.876686351" watchObservedRunningTime="2026-07-28 12:54:54.840502232 +0000 UTC m=+64.877871141"
Jul 28 12:54:54.918985 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:54.918950    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:56.016982 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.016950    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bf495b5d4-v98jh"]
Jul 28 12:54:56.020353 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.020328    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.023426 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.023402    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 12:54:56.024845 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.024820    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 12:54:56.024961 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.024847    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6wnn7\""
Jul 28 12:54:56.025266 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.025176    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 12:54:56.025366 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.025295    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 12:54:56.025559 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.025541    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9so4tk000c6ui\""
Jul 28 12:54:56.030955 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.030886    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bf495b5d4-v98jh"]
Jul 28 12:54:56.126369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.126250    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:56.126369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.126358    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afed194-561e-454b-a632-187cd39d4a22-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.126757 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.126417    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6afed194-561e-454b-a632-187cd39d4a22-client-ca-bundle\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.126757 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.126482    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6afed194-561e-454b-a632-187cd39d4a22-metrics-server-audit-profiles\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.126757 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.126523    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6afed194-561e-454b-a632-187cd39d4a22-secret-metrics-server-tls\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.126757 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.126556    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6afed194-561e-454b-a632-187cd39d4a22-audit-log\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.126757 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.126588    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6afed194-561e-454b-a632-187cd39d4a22-secret-metrics-server-client-certs\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.126757 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.126626    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mns7\" (UniqueName: \"kubernetes.io/projected/6afed194-561e-454b-a632-187cd39d4a22-kube-api-access-9mns7\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.131220 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.131196    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:54:56.139938 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.139911    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2058fd10-4ec4-45cd-b41f-3496159fa93c-original-pull-secret\") pod \"global-pull-secret-syncer-2f4qz\" (UID: \"2058fd10-4ec4-45cd-b41f-3496159fa93c\") " pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:56.214073 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:56.214051    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fad7fd_e532_475d_b891_db67a414256c.slice/crio-cb9d1e9dd11209d0ff12adb5bff58140f6fe500d33d374a209c06a48321c0bad WatchSource:0}: Error finding container cb9d1e9dd11209d0ff12adb5bff58140f6fe500d33d374a209c06a48321c0bad: Status 404 returned error can't find the container with id cb9d1e9dd11209d0ff12adb5bff58140f6fe500d33d374a209c06a48321c0bad
Jul 28 12:54:56.227940 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.227911    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afed194-561e-454b-a632-187cd39d4a22-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.228062 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.227971    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6afed194-561e-454b-a632-187cd39d4a22-client-ca-bundle\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.228062 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.228001    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:56.228233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.228208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6afed194-561e-454b-a632-187cd39d4a22-metrics-server-audit-profiles\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.228306 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.228283    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6afed194-561e-454b-a632-187cd39d4a22-secret-metrics-server-tls\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.228362 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.228331    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6afed194-561e-454b-a632-187cd39d4a22-audit-log\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.228412 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.228367    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6afed194-561e-454b-a632-187cd39d4a22-secret-metrics-server-client-certs\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.228465 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.228429    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mns7\" (UniqueName: \"kubernetes.io/projected/6afed194-561e-454b-a632-187cd39d4a22-kube-api-access-9mns7\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.228641 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.228611    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afed194-561e-454b-a632-187cd39d4a22-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.228931 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.228911    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6afed194-561e-454b-a632-187cd39d4a22-audit-log\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.229183 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.229161    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6afed194-561e-454b-a632-187cd39d4a22-metrics-server-audit-profiles\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.230979 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.230936    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6afed194-561e-454b-a632-187cd39d4a22-client-ca-bundle\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.231063 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.231049    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:54:56.231110 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.231072    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6afed194-561e-454b-a632-187cd39d4a22-secret-metrics-server-tls\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.231110 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.231096    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6afed194-561e-454b-a632-187cd39d4a22-secret-metrics-server-client-certs\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.236865 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.236843    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mns7\" (UniqueName: \"kubernetes.io/projected/6afed194-561e-454b-a632-187cd39d4a22-kube-api-access-9mns7\") pod \"metrics-server-bf495b5d4-v98jh\" (UID: \"6afed194-561e-454b-a632-187cd39d4a22\") " pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.242848 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.242823    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bece3c2-4eb4-4219-a757-45a57c351842-metrics-certs\") pod \"network-metrics-daemon-qrq22\" (UID: \"1bece3c2-4eb4-4219-a757-45a57c351842\") " pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:56.279573 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.279552    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4qz"
Jul 28 12:54:56.287644 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.287431    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z7m8f\""
Jul 28 12:54:56.294133 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.294102    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qrq22"
Jul 28 12:54:56.329731 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.329614    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:56.333773 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.333262    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:54:56.333773 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.333571    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:54:56.344595 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.344569    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:54:56.347291 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.346992    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"]
Jul 28 12:54:56.355329 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.353373    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:54:56.355329 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.355287    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:56.359656 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.359304    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbld7\" (UniqueName: \"kubernetes.io/projected/2ebe21a0-6999-409b-a63b-0fd0265d25b5-kube-api-access-zbld7\") pod \"network-check-target-cqxxn\" (UID: \"2ebe21a0-6999-409b-a63b-0fd0265d25b5\") " pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:56.360343 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.360124    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dc24h\""
Jul 28 12:54:56.367555 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.367351    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 12:54:56.375060 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.374976    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"]
Jul 28 12:54:56.389134 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.389115    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kmztt\""
Jul 28 12:54:56.397646 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.397446    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:54:56.432158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.431926    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aac227a6-52b5-41f6-8cce-f83aac850e25-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-g9mcw\" (UID: \"aac227a6-52b5-41f6-8cce-f83aac850e25\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:54:56.432158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.432014    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gq74\" (UniqueName: \"kubernetes.io/projected/aac227a6-52b5-41f6-8cce-f83aac850e25-kube-api-access-6gq74\") pod \"monitoring-plugin-68b7fc9574-g9mcw\" (UID: \"aac227a6-52b5-41f6-8cce-f83aac850e25\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:54:56.441522 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.441459    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:56.507267 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.507184    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2f4qz"]
Jul 28 12:54:56.533307 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.532859    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aac227a6-52b5-41f6-8cce-f83aac850e25-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-g9mcw\" (UID: \"aac227a6-52b5-41f6-8cce-f83aac850e25\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:54:56.533307 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.532937    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gq74\" (UniqueName: \"kubernetes.io/projected/aac227a6-52b5-41f6-8cce-f83aac850e25-kube-api-access-6gq74\") pod \"monitoring-plugin-68b7fc9574-g9mcw\" (UID: \"aac227a6-52b5-41f6-8cce-f83aac850e25\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:54:56.536507 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.536270    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aac227a6-52b5-41f6-8cce-f83aac850e25-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-g9mcw\" (UID: \"aac227a6-52b5-41f6-8cce-f83aac850e25\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:54:56.538784 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.538755    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qrq22"]
Jul 28 12:54:56.545868 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.545846    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gq74\" (UniqueName: \"kubernetes.io/projected/aac227a6-52b5-41f6-8cce-f83aac850e25-kube-api-access-6gq74\") pod \"monitoring-plugin-68b7fc9574-g9mcw\" (UID: \"aac227a6-52b5-41f6-8cce-f83aac850e25\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:54:56.570426 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.569612    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bf495b5d4-v98jh"]
Jul 28 12:54:56.572689 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:56.572663    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afed194_561e_454b_a632_187cd39d4a22.slice/crio-891716e27563e083a07724ce6fc8837da7d37a216715829af0e63ad0effcafe9 WatchSource:0}: Error finding container 891716e27563e083a07724ce6fc8837da7d37a216715829af0e63ad0effcafe9: Status 404 returned error can't find the container with id 891716e27563e083a07724ce6fc8837da7d37a216715829af0e63ad0effcafe9
Jul 28 12:54:56.593724 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.593656    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cqxxn"]
Jul 28 12:54:56.596057 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:56.596036    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ebe21a0_6999_409b_a63b_0fd0265d25b5.slice/crio-038d3949fcebedcbad3340c8bc52ea266683cb3fe1705e68bab18fa055c80d1a WatchSource:0}: Error finding container 038d3949fcebedcbad3340c8bc52ea266683cb3fe1705e68bab18fa055c80d1a: Status 404 returned error can't find the container with id 038d3949fcebedcbad3340c8bc52ea266683cb3fe1705e68bab18fa055c80d1a
Jul 28 12:54:56.680142 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.680109    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:54:56.813895 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.813869    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"]
Jul 28 12:54:56.816357 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:54:56.816204    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac227a6_52b5_41f6_8cce_f83aac850e25.slice/crio-b68072de49df335839059f2514b4964c0b5906f16bee5b09e1d3714964a297fa WatchSource:0}: Error finding container b68072de49df335839059f2514b4964c0b5906f16bee5b09e1d3714964a297fa: Status 404 returned error can't find the container with id b68072de49df335839059f2514b4964c0b5906f16bee5b09e1d3714964a297fa
Jul 28 12:54:56.828298 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.828262    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-td7h2" event={"ID":"59199dc1-6c80-47c3-a6a4-fb9bcef71268","Type":"ContainerStarted","Data":"7ae08d139847cf927a6dcf33eed5ab70fc08524ec7c44b74b495138191f37068"}
Jul 28 12:54:56.830201 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.830175    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5" event={"ID":"6a73e5a1-f9e8-4c93-9790-b1315e823e13","Type":"ContainerStarted","Data":"84b2985c4fe43c64e3134e605512e091ae7ca71ba39b7cc848228cd901593014"}
Jul 28 12:54:56.830283 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.830223    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5" event={"ID":"6a73e5a1-f9e8-4c93-9790-b1315e823e13","Type":"ContainerStarted","Data":"5005418295cfb1903b915c0c50e0fd49ac2f026530b0327c645d3d0f7c956f86"}
Jul 28 12:54:56.830283 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.830246    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5" event={"ID":"6a73e5a1-f9e8-4c93-9790-b1315e823e13","Type":"ContainerStarted","Data":"2b095642d9f15c917c8fd239ee88fd85caaa4ae64a6001f1c14c8d9b4ac2d2c4"}
Jul 28 12:54:56.831371 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.831354    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh" event={"ID":"6afed194-561e-454b-a632-187cd39d4a22","Type":"ContainerStarted","Data":"891716e27563e083a07724ce6fc8837da7d37a216715829af0e63ad0effcafe9"}
Jul 28 12:54:56.832291 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.832263    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2f4qz" event={"ID":"2058fd10-4ec4-45cd-b41f-3496159fa93c","Type":"ContainerStarted","Data":"7822201a0a66f62334b5dc7bf373140ed72bbdff3d47c1ca060ca4396776c91a"}
Jul 28 12:54:56.833222 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.833203    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz" event={"ID":"19fad7fd-e532-475d-b891-db67a414256c","Type":"ContainerStarted","Data":"cb9d1e9dd11209d0ff12adb5bff58140f6fe500d33d374a209c06a48321c0bad"}
Jul 28 12:54:56.834554 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.834526    2553 generic.go:356] "Generic (PLEG): container finished" podID="3f930747-3471-4519-afeb-6620221b95a9" containerID="fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940" exitCode=0
Jul 28 12:54:56.834615 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.834595    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerDied","Data":"fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940"}
Jul 28 12:54:56.835815 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.835777    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw" event={"ID":"aac227a6-52b5-41f6-8cce-f83aac850e25","Type":"ContainerStarted","Data":"b68072de49df335839059f2514b4964c0b5906f16bee5b09e1d3714964a297fa"}
Jul 28 12:54:56.837019 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.837002    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qrq22" event={"ID":"1bece3c2-4eb4-4219-a757-45a57c351842","Type":"ContainerStarted","Data":"5879d4af04c55fd163cc0042af6a1cfb454f44cdf311c85e4968cdd1dc9d4fc7"}
Jul 28 12:54:56.838537 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.838520    2553 generic.go:356] "Generic (PLEG): container finished" podID="addd1058-8188-4cb5-b23a-e25a8dfb393c" containerID="2f4f9e5d6493d7c568ddbc05ad3fc2ce22d89f5fb3d8835a44d5848f5fb97ef5" exitCode=0
Jul 28 12:54:56.838648 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.838575    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6drlp" event={"ID":"addd1058-8188-4cb5-b23a-e25a8dfb393c","Type":"ContainerDied","Data":"2f4f9e5d6493d7c568ddbc05ad3fc2ce22d89f5fb3d8835a44d5848f5fb97ef5"}
Jul 28 12:54:56.839786 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.839766    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cqxxn" event={"ID":"2ebe21a0-6999-409b-a63b-0fd0265d25b5","Type":"ContainerStarted","Data":"038d3949fcebedcbad3340c8bc52ea266683cb3fe1705e68bab18fa055c80d1a"}
Jul 28 12:54:56.851737 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.851704    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-td7h2" podStartSLOduration=1.557789936 podStartE2EDuration="14.851691732s" totalImagesPullingTime="13.293901796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:54:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:56.850329596 +0000 UTC m=+66.887698530" watchObservedRunningTime="2026-07-28 12:54:56.851691732 +0000 UTC m=+66.889060630"
Jul 28 12:54:56.899035 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:56.898992    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-kp9k5" podStartSLOduration=4.038571749 podStartE2EDuration="5.898977078s" totalImagesPullingTime="1.860405329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:56.870117095 +0000 UTC m=+66.907486006" watchObservedRunningTime="2026-07-28 12:54:56.898977078 +0000 UTC m=+66.936345983"
Jul 28 12:54:57.316787 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.316703    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-cn6x5"]
Jul 28 12:54:57.339933 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.339885    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-cn6x5"]
Jul 28 12:54:57.340128 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.340006    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-cn6x5"
Jul 28 12:54:57.344524 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.344320    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:54:57.345749 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.345543    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6nkhm\""
Jul 28 12:54:57.345749 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.345547    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:54:57.346627 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.346541    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:57.440138 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.440017    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bl45\" (UniqueName: \"kubernetes.io/projected/111c52cc-e449-4068-b7ed-a1af6ad8b5f1-kube-api-access-7bl45\") pod \"downloads-7c97dc4555-cn6x5\" (UID: \"111c52cc-e449-4068-b7ed-a1af6ad8b5f1\") " pod="openshift-console/downloads-7c97dc4555-cn6x5"
Jul 28 12:54:57.446553 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.446462    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:57.512389 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.512240    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:57.541992 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.541551    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bl45\" (UniqueName: \"kubernetes.io/projected/111c52cc-e449-4068-b7ed-a1af6ad8b5f1-kube-api-access-7bl45\") pod \"downloads-7c97dc4555-cn6x5\" (UID: \"111c52cc-e449-4068-b7ed-a1af6ad8b5f1\") " pod="openshift-console/downloads-7c97dc4555-cn6x5"
Jul 28 12:54:57.554982 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.554885    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bl45\" (UniqueName: \"kubernetes.io/projected/111c52cc-e449-4068-b7ed-a1af6ad8b5f1-kube-api-access-7bl45\") pod \"downloads-7c97dc4555-cn6x5\" (UID: \"111c52cc-e449-4068-b7ed-a1af6ad8b5f1\") " pod="openshift-console/downloads-7c97dc4555-cn6x5"
Jul 28 12:54:57.631073 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.630966    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:57.659245 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.658908    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-cn6x5"
Jul 28 12:54:57.831237 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.831178    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:57.851587 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.850867    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6drlp" event={"ID":"addd1058-8188-4cb5-b23a-e25a8dfb393c","Type":"ContainerStarted","Data":"d94a4174a966fb6f28efb370c16b2934862b4d97a19b245dcd86ee3b9baa1d90"}
Jul 28 12:54:57.851587 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.850923    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6drlp" event={"ID":"addd1058-8188-4cb5-b23a-e25a8dfb393c","Type":"ContainerStarted","Data":"0bcd118cda9b5d087adb922d7af4144e80604988690e94b5a610041df9653171"}
Jul 28 12:54:57.875268 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.874570    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6drlp" podStartSLOduration=5.033684282 podStartE2EDuration="6.87454735s" totalImagesPullingTime="1.671488641s" totalInitContainerRuntime="169.374427ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:57.873756626 +0000 UTC m=+67.911125534" watchObservedRunningTime="2026-07-28 12:54:57.87454735 +0000 UTC m=+67.911916258"
Jul 28 12:54:57.992148 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:57.992089    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:58.008997 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.008960    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:54:58.031627 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.030976    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:54:58.037386 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.031418    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.047827 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.047651    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 12:54:58.051347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.048136    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 12:54:58.051347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.048424    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 12:54:58.051347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.049461    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 12:54:58.051347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.049947    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 12:54:58.051347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.050196    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 12:54:58.051347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.050460    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f29cs\""
Jul 28 12:54:58.051347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.050709    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 12:54:58.051347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.050974    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 12:54:58.051347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.051172    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fuor46sueeiqf\""
Jul 28 12:54:58.051843 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.051493    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 12:54:58.051843 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.051679    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 12:54:58.052530 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.052042    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 12:54:58.052530 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.052305    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 12:54:58.062686 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.062660    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 12:54:58.065195 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.062663    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.147951    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.147997    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148036    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148064    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148094    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-config\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148118    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148152    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-web-config\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148185    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148209    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-config-out\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148230    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148265    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148294    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148341    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smlv4\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-kube-api-access-smlv4\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148369    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148392    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.148734 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148433    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.149605 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148465    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.149605 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148490    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.149605 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148515    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.149605 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.148547    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.210233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.209852    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249187    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249237    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249266    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249296    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-config\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249320    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249367    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-web-config\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249404    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249429    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-config-out\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249454    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249491    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249520    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249574    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smlv4\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-kube-api-access-smlv4\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249604    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249633    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249666    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249698    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.250508 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249721    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.251523 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249749    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.251523 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249784    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.251523 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.249827    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.251523 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.250365    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.251523 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.250453    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.251523 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.250946    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.252243 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.251999    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.252563 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.252462    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.253419 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.252919    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-web-config\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.253851 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.253822    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.254896 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.254481    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.254896 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.254526    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.257325 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.257286    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.258303 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.258226    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-config-out\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.258395 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.258352    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.260478 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.259968    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.260478 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.260408    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.260623 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.260503    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.260623 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.260530    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.263374 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.262927    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.267552 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.267333    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-config\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.267735 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.267679    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smlv4\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-kube-api-access-smlv4\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.267735 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.267694    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.313095 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.313005    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:58.367192 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.366861    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:54:58.415367 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.415300    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:58.484260 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.484197    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:58.745048 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:58.744962    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:59.593515 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:59.593469    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:59.650866 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:59.650815    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:54:59.794238 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:54:59.794204    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2hlrq"
Jul 28 12:55:00.248244 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:00.248146    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:00.322555 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:00.322516    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:01.303924 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:01.303886    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:01.412826 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:01.409477    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:01.507758 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:01.507718    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-cn6x5"]
Jul 28 12:55:02.032699 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:02.032660    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:02.097562 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:02.097518    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:02.747411 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:55:02.747365    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111c52cc_e449_4068_b7ed_a1af6ad8b5f1.slice/crio-a0e67f29f6dab7efde9483502eb345477b0dcfabc7c74d92df67139f1398d7d9 WatchSource:0}: Error finding container a0e67f29f6dab7efde9483502eb345477b0dcfabc7c74d92df67139f1398d7d9: Status 404 returned error can't find the container with id a0e67f29f6dab7efde9483502eb345477b0dcfabc7c74d92df67139f1398d7d9
Jul 28 12:55:02.878329 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:02.878301    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-cn6x5" event={"ID":"111c52cc-e449-4068-b7ed-a1af6ad8b5f1","Type":"ContainerStarted","Data":"a0e67f29f6dab7efde9483502eb345477b0dcfabc7c74d92df67139f1398d7d9"}
Jul 28 12:55:02.909442 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:02.909411    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:55:03.201529 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.201478    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:03.302168 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.302139    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:03.886008 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.885979    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerStarted","Data":"278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368"}
Jul 28 12:55:03.886394 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.886082    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerStarted","Data":"b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e"}
Jul 28 12:55:03.886394 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.886118    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerStarted","Data":"2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55"}
Jul 28 12:55:03.886394 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.886153    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerStarted","Data":"4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae"}
Jul 28 12:55:03.886394 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.886177    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerStarted","Data":"b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99"}
Jul 28 12:55:03.887788 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.887699    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw" event={"ID":"aac227a6-52b5-41f6-8cce-f83aac850e25","Type":"ContainerStarted","Data":"736e285ef2605b05964eebe743e32410665f3e4391db2169ed23182e4e6ce6ae"}
Jul 28 12:55:03.887917 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.887839    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:55:03.890645 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.890616    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qrq22" event={"ID":"1bece3c2-4eb4-4219-a757-45a57c351842","Type":"ContainerStarted","Data":"31c57390761d8115696ec1edd57d1945e6d467d7a6f18ff460ccc710bd4bbf2a"}
Jul 28 12:55:03.890738 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.890670    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qrq22" event={"ID":"1bece3c2-4eb4-4219-a757-45a57c351842","Type":"ContainerStarted","Data":"7ec623121663b7fdd6303ab8ed18bba7cda4a6ba598b034741c70b94292ba016"}
Jul 28 12:55:03.893079 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.893034    2553 generic.go:356] "Generic (PLEG): container finished" podID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerID="edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d" exitCode=0
Jul 28 12:55:03.893183 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.893142    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerDied","Data":"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"}
Jul 28 12:55:03.893261 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.893236    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerStarted","Data":"19f9f1d5327c2d9409134e677d7204bc1475f883b9291798f71f19a1350c602b"}
Jul 28 12:55:03.894093 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.894074    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw"
Jul 28 12:55:03.895917 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.895877    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cqxxn" event={"ID":"2ebe21a0-6999-409b-a63b-0fd0265d25b5","Type":"ContainerStarted","Data":"73d1c451aefa63bcc2c1c15320a427994655bd178cbf8477e5c18e1ca8c6c7f3"}
Jul 28 12:55:03.896022 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.895975    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:55:03.898553 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.898527    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh" event={"ID":"6afed194-561e-454b-a632-187cd39d4a22","Type":"ContainerStarted","Data":"2669ffa2b3402de93946a0eb6b00e171a79694729f145628324f67560035a20f"}
Jul 28 12:55:03.900226 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.900203    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2f4qz" event={"ID":"2058fd10-4ec4-45cd-b41f-3496159fa93c","Type":"ContainerStarted","Data":"4b0fbe0ce7772cc7b0024fb64ba663faa1650a9cf20513011d2f29ec353c1d3a"}
Jul 28 12:55:03.902994 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.902975    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz" event={"ID":"19fad7fd-e532-475d-b891-db67a414256c","Type":"ContainerStarted","Data":"64b96174bda8ce57aea2e4da6b786c88bccedde6f9874f6dd5785e4efdc12beb"}
Jul 28 12:55:03.903082 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.903035    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz" event={"ID":"19fad7fd-e532-475d-b891-db67a414256c","Type":"ContainerStarted","Data":"462df4e81f68abfecce9c6e1228c12e65c84d9c94a8209880b0aa32041182175"}
Jul 28 12:55:03.903082 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.903076    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz" event={"ID":"19fad7fd-e532-475d-b891-db67a414256c","Type":"ContainerStarted","Data":"41c2ff5a955891116c28a20f8ab68640c79f95a8e1699e67c2645d9ba6588180"}
Jul 28 12:55:03.906601 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.906519    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-g9mcw" podStartSLOduration=1.975942303 podStartE2EDuration="7.906507032s" totalImagesPullingTime="5.930564729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:03.905093138 +0000 UTC m=+73.942462048" watchObservedRunningTime="2026-07-28 12:55:03.906507032 +0000 UTC m=+73.943875939"
Jul 28 12:55:03.924514 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.924472    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qrq22" podStartSLOduration=69.145104851 podStartE2EDuration="1m13.924456195s" totalImagesPullingTime="4.779351344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:03.923662467 +0000 UTC m=+73.961031374" watchObservedRunningTime="2026-07-28 12:55:03.924456195 +0000 UTC m=+73.961825102"
Jul 28 12:55:03.945123 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.945078    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh" podStartSLOduration=2.769790661 podStartE2EDuration="8.945062903s" totalImagesPullingTime="6.175272242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:03.944145165 +0000 UTC m=+73.981514065" watchObservedRunningTime="2026-07-28 12:55:03.945062903 +0000 UTC m=+73.982431811"
Jul 28 12:55:03.989060 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:03.989012    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2f4qz" podStartSLOduration=67.719816088 podStartE2EDuration="1m13.988994105s" totalImagesPullingTime="6.269178017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:03.98745563 +0000 UTC m=+74.024824537" watchObservedRunningTime="2026-07-28 12:55:03.988994105 +0000 UTC m=+74.026363013"
Jul 28 12:55:04.009343 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.009292    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cqxxn" podStartSLOduration=67.857812253 podStartE2EDuration="1m14.009271889s" totalImagesPullingTime="6.151459636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:04.006585306 +0000 UTC m=+74.043954215" watchObservedRunningTime="2026-07-28 12:55:04.009271889 +0000 UTC m=+74.046640799"
Jul 28 12:55:04.027363 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.027311    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:04.118404 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.118336    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:04.186276 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.186194    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:04.913250 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.913144    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz" event={"ID":"19fad7fd-e532-475d-b891-db67a414256c","Type":"ContainerStarted","Data":"3326f01292ddf4000495f182bf6f5ee41aef7a4e72f706bdcb559a60da431bfa"}
Jul 28 12:55:04.913697 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.913267    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz" event={"ID":"19fad7fd-e532-475d-b891-db67a414256c","Type":"ContainerStarted","Data":"2c1dc88d2e1be955dbaf9bcfdf6dd6bca5453a4fb243c28258fe06ced2081b7c"}
Jul 28 12:55:04.913697 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.913317    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz" event={"ID":"19fad7fd-e532-475d-b891-db67a414256c","Type":"ContainerStarted","Data":"3b30af9bd884713772388c0e90297e0f4b6bcc063b0c9aacab3adac78374c8ad"}
Jul 28 12:55:04.913957 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.913878    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:55:04.918401 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.918375    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerStarted","Data":"b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4"}
Jul 28 12:55:04.940577 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.940527    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz" podStartSLOduration=4.697837382 podStartE2EDuration="11.940510903s" totalImagesPullingTime="7.242673521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:04.938415793 +0000 UTC m=+74.975784706" watchObservedRunningTime="2026-07-28 12:55:04.940510903 +0000 UTC m=+74.977879811"
Jul 28 12:55:04.968320 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:04.968265    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.6171890810000003 podStartE2EDuration="12.968248406s" totalImagesPullingTime="10.301759664s" totalInitContainerRuntime="49.299661ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:04.965380438 +0000 UTC m=+75.002749346" watchObservedRunningTime="2026-07-28 12:55:04.968248406 +0000 UTC m=+75.005617360"
Jul 28 12:55:06.176579 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:06.176539    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:06.264725 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:06.264687    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:07.947085 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:07.946746    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerStarted","Data":"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"}
Jul 28 12:55:07.947085 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:07.946892    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerStarted","Data":"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"}
Jul 28 12:55:07.947085 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:07.946936    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerStarted","Data":"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"}
Jul 28 12:55:07.947085 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:07.946976    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerStarted","Data":"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"}
Jul 28 12:55:08.957351 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:08.957311    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerStarted","Data":"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"}
Jul 28 12:55:08.957780 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:08.957384    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerStarted","Data":"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"}
Jul 28 12:55:08.976261 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:08.976227    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:09.000648 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:09.000584    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=8.279479824 podStartE2EDuration="12.000560825s" totalImagesPullingTime="3.68635838s" totalInitContainerRuntime="34.722621ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:08.997764559 +0000 UTC m=+79.035133520" watchObservedRunningTime="2026-07-28 12:55:09.000560825 +0000 UTC m=+79.037929734"
Jul 28 12:55:09.074310 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:09.074270    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:10.931833 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:10.931781    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c8fdfcf97-7h2kz"
Jul 28 12:55:11.114113 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:11.114059    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:11.215466 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:11.215379    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:12.442401 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:12.442269    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:12.510288 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:12.510250    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:13.367902 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:13.367870    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:55:15.839568 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:15.839532    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:15.898304 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:15.898242    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:16.335164 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:16.335117    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:55:16.335348 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:16.335193    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:55:18.201943 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:18.201904    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:18.271985 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:18.271937    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:19.236471 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:19.236434    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:19.298045 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:19.298005    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:20.431035 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:20.431001    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:20.516779 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:20.516742    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:22.022166 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:22.022125    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-cn6x5" event={"ID":"111c52cc-e449-4068-b7ed-a1af6ad8b5f1","Type":"ContainerStarted","Data":"6bd980c5993a5dc7cc4ec1bd99d53deaa763177eaf3b8f18f1057e0c26bb1e50"}
Jul 28 12:55:22.022671 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:22.022332    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-cn6x5"
Jul 28 12:55:22.035663 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:22.035633    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-cn6x5"
Jul 28 12:55:22.062856 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:22.062769    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-cn6x5" podStartSLOduration=6.720135985 podStartE2EDuration="25.062747819s" totalImagesPullingTime="18.342611834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:22.042416414 +0000 UTC m=+92.079785322" watchObservedRunningTime="2026-07-28 12:55:22.062747819 +0000 UTC m=+92.100116731"
Jul 28 12:55:24.054258 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:24.054209    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:24.122303 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:24.122264    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:26.789482 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:26.789445    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:26.900522 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:26.900485    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:27.000242 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:27.000210    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:31.985344 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:31.985295    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:32.062704 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:32.062673    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:33.204694 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:33.204640    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:33.269292 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:33.269262    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:34.921376 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:34.921343    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cqxxn"
Jul 28 12:55:36.340198 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:36.340167    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:55:36.344086 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:36.344060    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bf495b5d4-v98jh"
Jul 28 12:55:43.393118 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:43.393075    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:55:58.368051 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:58.368007    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:55:58.417047 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:58.417013    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:55:59.201679 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:55:59.201645    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:10.930385 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:10.930339    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:11.368928 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:11.368831    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:11.997119 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:11.997078    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:56:12.232806 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:12.232766    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="alertmanager" containerID="cri-o://b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99" gracePeriod=120
Jul 28 12:56:12.233002 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:12.232925    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy" containerID="cri-o://b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e" gracePeriod=120
Jul 28 12:56:12.233002 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:12.232945    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy-web" containerID="cri-o://2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55" gracePeriod=120
Jul 28 12:56:12.233002 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:12.232924    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="prom-label-proxy" containerID="cri-o://b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4" gracePeriod=120
Jul 28 12:56:12.233169 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:12.232986    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy-metric" containerID="cri-o://278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368" gracePeriod=120
Jul 28 12:56:12.233169 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:12.232999    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="config-reloader" containerID="cri-o://4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae" gracePeriod=120
Jul 28 12:56:13.239853 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.239814    2553 generic.go:356] "Generic (PLEG): container finished" podID="3f930747-3471-4519-afeb-6620221b95a9" containerID="b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4" exitCode=0
Jul 28 12:56:13.239853 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.239846    2553 generic.go:356] "Generic (PLEG): container finished" podID="3f930747-3471-4519-afeb-6620221b95a9" containerID="278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368" exitCode=0
Jul 28 12:56:13.239853 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.239853    2553 generic.go:356] "Generic (PLEG): container finished" podID="3f930747-3471-4519-afeb-6620221b95a9" containerID="b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e" exitCode=0
Jul 28 12:56:13.239853 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.239859    2553 generic.go:356] "Generic (PLEG): container finished" podID="3f930747-3471-4519-afeb-6620221b95a9" containerID="4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae" exitCode=0
Jul 28 12:56:13.239853 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.239864    2553 generic.go:356] "Generic (PLEG): container finished" podID="3f930747-3471-4519-afeb-6620221b95a9" containerID="b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99" exitCode=0
Jul 28 12:56:13.240342 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.239820    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerDied","Data":"b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4"}
Jul 28 12:56:13.240342 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.239944    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerDied","Data":"278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368"}
Jul 28 12:56:13.240342 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.239973    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerDied","Data":"b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e"}
Jul 28 12:56:13.240342 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.239997    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerDied","Data":"4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae"}
Jul 28 12:56:13.240342 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.240020    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerDied","Data":"b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99"}
Jul 28 12:56:13.476040 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.476012    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:13.521531 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521458    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-main-db\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.521531 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521506    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-main-tls\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.521531 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521527    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-tls-assets\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.521783 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521568    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.521783 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521598    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.521783 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521630    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km72t\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-kube-api-access-km72t\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.521783 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521680    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-cluster-tls-config\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.521783 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521722    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-trusted-ca-bundle\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.521783 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521758    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-config-volume\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.521783 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521780    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-metrics-client-ca\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.522165 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521833    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-config-out\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.522165 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521888    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.522165 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521880    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:56:13.522165 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.521911    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-web-config\") pod \"3f930747-3471-4519-afeb-6620221b95a9\" (UID: \"3f930747-3471-4519-afeb-6620221b95a9\") "
Jul 28 12:56:13.522369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.522262    2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-main-db\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.522369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.522268    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:13.522369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.522336    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:13.526996 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.526861    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:13.527122 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.526963    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-config-out" (OuterVolumeSpecName: "config-out") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:56:13.527122 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.527058    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:56:13.527402 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.527366    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-config-volume" (OuterVolumeSpecName: "config-volume") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:13.527711 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.527579    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-kube-api-access-km72t" (OuterVolumeSpecName: "kube-api-access-km72t") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "kube-api-access-km72t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:56:13.527845 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.527684    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:13.528113 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.528091    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:13.528920 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.528887    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:13.532918 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.532894    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:13.541966 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.541937    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-web-config" (OuterVolumeSpecName: "web-config") pod "3f930747-3471-4519-afeb-6620221b95a9" (UID: "3f930747-3471-4519-afeb-6620221b95a9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:13.623105 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623064    2553 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-cluster-tls-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623105 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623099    2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623105 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623111    2553 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-config-volume\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623121    2553 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f930747-3471-4519-afeb-6620221b95a9-metrics-client-ca\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623132    2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f930747-3471-4519-afeb-6620221b95a9-config-out\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623140    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623148    2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-web-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623156    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-main-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623164    2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-tls-assets\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623174    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623183    2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f930747-3471-4519-afeb-6620221b95a9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:13.623334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:13.623192    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km72t\" (UniqueName: \"kubernetes.io/projected/3f930747-3471-4519-afeb-6620221b95a9-kube-api-access-km72t\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:14.246788 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.246755    2553 generic.go:356] "Generic (PLEG): container finished" podID="3f930747-3471-4519-afeb-6620221b95a9" containerID="2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55" exitCode=0
Jul 28 12:56:14.247188 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.246832    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerDied","Data":"2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55"}
Jul 28 12:56:14.247188 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.246867    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.247188 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.246921    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f930747-3471-4519-afeb-6620221b95a9","Type":"ContainerDied","Data":"4e861a47f909bcd12f82c6aeb67f400a68760437b51309372e479539c1ae45cc"}
Jul 28 12:56:14.247188 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.246970    2553 scope.go:117] "RemoveContainer" containerID="b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4"
Jul 28 12:56:14.255145 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.255014    2553 scope.go:117] "RemoveContainer" containerID="278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368"
Jul 28 12:56:14.262616 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.262598    2553 scope.go:117] "RemoveContainer" containerID="b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e"
Jul 28 12:56:14.269141 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.269123    2553 scope.go:117] "RemoveContainer" containerID="2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55"
Jul 28 12:56:14.273150 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.273087    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:56:14.277633 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.277591    2553 scope.go:117] "RemoveContainer" containerID="4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae"
Jul 28 12:56:14.279092 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.279065    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:56:14.284704 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.284686    2553 scope.go:117] "RemoveContainer" containerID="b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99"
Jul 28 12:56:14.291762 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.291742    2553 scope.go:117] "RemoveContainer" containerID="fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940"
Jul 28 12:56:14.298639 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.298617    2553 scope.go:117] "RemoveContainer" containerID="b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4"
Jul 28 12:56:14.298920 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:14.298900    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4\": container with ID starting with b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4 not found: ID does not exist" containerID="b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4"
Jul 28 12:56:14.298976 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.298929    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4"} err="failed to get container status \"b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4\": rpc error: code = NotFound desc = could not find container \"b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4\": container with ID starting with b9a3c0e372d4dc90720eb3c3949379491aa5f4a42b178b134b679281b12a75f4 not found: ID does not exist"
Jul 28 12:56:14.298976 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.298965    2553 scope.go:117] "RemoveContainer" containerID="278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368"
Jul 28 12:56:14.299171 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:14.299156    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368\": container with ID starting with 278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368 not found: ID does not exist" containerID="278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368"
Jul 28 12:56:14.299329 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.299173    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368"} err="failed to get container status \"278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368\": rpc error: code = NotFound desc = could not find container \"278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368\": container with ID starting with 278046bcc63633d1469fc6dfad1b4ca7f51ed9eab968db76e360d9b1c5dd8368 not found: ID does not exist"
Jul 28 12:56:14.299329 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.299184    2553 scope.go:117] "RemoveContainer" containerID="b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e"
Jul 28 12:56:14.299492 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:14.299428    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e\": container with ID starting with b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e not found: ID does not exist" containerID="b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e"
Jul 28 12:56:14.299492 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.299450    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e"} err="failed to get container status \"b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e\": rpc error: code = NotFound desc = could not find container \"b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e\": container with ID starting with b55c1380f9761c644e31987308d865e0e07d0379f351c8999a3a030b46495c0e not found: ID does not exist"
Jul 28 12:56:14.299492 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.299470    2553 scope.go:117] "RemoveContainer" containerID="2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55"
Jul 28 12:56:14.299954 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:14.299746    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55\": container with ID starting with 2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55 not found: ID does not exist" containerID="2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55"
Jul 28 12:56:14.299954 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.299781    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55"} err="failed to get container status \"2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55\": rpc error: code = NotFound desc = could not find container \"2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55\": container with ID starting with 2a7e16cf75cb41c135ad2c506bca57b62f5131d942f186193adf2dc943118c55 not found: ID does not exist"
Jul 28 12:56:14.299954 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.299819    2553 scope.go:117] "RemoveContainer" containerID="4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae"
Jul 28 12:56:14.300354 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:14.300114    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae\": container with ID starting with 4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae not found: ID does not exist" containerID="4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae"
Jul 28 12:56:14.300354 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.300142    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae"} err="failed to get container status \"4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae\": rpc error: code = NotFound desc = could not find container \"4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae\": container with ID starting with 4ea2f677f8d01eddde1b13e17ff1d9d42aa12f35f59e18feb49edfb7eefca0ae not found: ID does not exist"
Jul 28 12:56:14.300354 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.300168    2553 scope.go:117] "RemoveContainer" containerID="b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99"
Jul 28 12:56:14.300492 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:14.300465    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99\": container with ID starting with b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99 not found: ID does not exist" containerID="b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99"
Jul 28 12:56:14.300492 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.300482    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99"} err="failed to get container status \"b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99\": rpc error: code = NotFound desc = could not find container \"b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99\": container with ID starting with b88c75565e04589f2d06acc834eb5c4fa27875e2e57af82bd7d894cf15f91b99 not found: ID does not exist"
Jul 28 12:56:14.300557 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.300496    2553 scope.go:117] "RemoveContainer" containerID="fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940"
Jul 28 12:56:14.300734 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:14.300716    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940\": container with ID starting with fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940 not found: ID does not exist" containerID="fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940"
Jul 28 12:56:14.300772 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.300739    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940"} err="failed to get container status \"fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940\": rpc error: code = NotFound desc = could not find container \"fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940\": container with ID starting with fbe2eb369350ca5678be728df92516814117407679d2e064925e28abc5392940 not found: ID does not exist"
Jul 28 12:56:14.313526 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313495    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:56:14.313872 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313843    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy-metric"
Jul 28 12:56:14.313872 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313859    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy-metric"
Jul 28 12:56:14.313872 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313872    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="init-config-reloader"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313878    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="init-config-reloader"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313885    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="prom-label-proxy"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313890    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="prom-label-proxy"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313897    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="alertmanager"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313902    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="alertmanager"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313909    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="config-reloader"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313915    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="config-reloader"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313924    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy-web"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313930    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy-web"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313938    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313944    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313990    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="alertmanager"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.313998    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="config-reloader"
Jul 28 12:56:14.314005 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.314005    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy-web"
Jul 28 12:56:14.314450 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.314014    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="prom-label-proxy"
Jul 28 12:56:14.314450 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.314021    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy"
Jul 28 12:56:14.314450 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.314028    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f930747-3471-4519-afeb-6620221b95a9" containerName="kube-rbac-proxy-metric"
Jul 28 12:56:14.317692 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.317674    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.321102 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.321083    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 12:56:14.321215 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.321102    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 12:56:14.321495 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.321474    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 12:56:14.321617 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.321477    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 12:56:14.321617 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.321545    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 12:56:14.321895 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.321867    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ptmgh\""
Jul 28 12:56:14.321975 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.321939    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 12:56:14.322042 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.321976    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 12:56:14.322248 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.322233    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 12:56:14.328873 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.328854    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 12:56:14.334673 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.334651    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:56:14.429478 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429437    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f15b3ac1-60d8-4b82-b686-9e52b6d23868-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429478 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429481    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429708 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429504    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429708 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429599    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429708 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429633    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f15b3ac1-60d8-4b82-b686-9e52b6d23868-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429708 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429654    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429708 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429673    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f15b3ac1-60d8-4b82-b686-9e52b6d23868-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429929 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429715    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-web-config\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429929 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429750    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-config-volume\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429929 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429764    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f15b3ac1-60d8-4b82-b686-9e52b6d23868-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429929 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429778    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429929 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429821    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2xw\" (UniqueName: \"kubernetes.io/projected/f15b3ac1-60d8-4b82-b686-9e52b6d23868-kube-api-access-fm2xw\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.429929 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.429873    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f15b3ac1-60d8-4b82-b686-9e52b6d23868-config-out\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.475268 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.475235    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f930747-3471-4519-afeb-6620221b95a9" path="/var/lib/kubelet/pods/3f930747-3471-4519-afeb-6620221b95a9/volumes"
Jul 28 12:56:14.530784 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.530700    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.530784 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.530741    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f15b3ac1-60d8-4b82-b686-9e52b6d23868-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.530784 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.530770    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531016 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.530824    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f15b3ac1-60d8-4b82-b686-9e52b6d23868-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531016 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.530885    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-web-config\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531084 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531018    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-config-volume\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531084 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531055    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f15b3ac1-60d8-4b82-b686-9e52b6d23868-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531181 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531084    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531181 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531108    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2xw\" (UniqueName: \"kubernetes.io/projected/f15b3ac1-60d8-4b82-b686-9e52b6d23868-kube-api-access-fm2xw\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531181 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531144    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f15b3ac1-60d8-4b82-b686-9e52b6d23868-config-out\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531323 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531169    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f15b3ac1-60d8-4b82-b686-9e52b6d23868-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531323 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531222    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.531323 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531257    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.532158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531589    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f15b3ac1-60d8-4b82-b686-9e52b6d23868-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.532158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.531951    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f15b3ac1-60d8-4b82-b686-9e52b6d23868-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.532331 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.532260    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f15b3ac1-60d8-4b82-b686-9e52b6d23868-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.534158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.534032    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-web-config\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.534158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.534095    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-config-volume\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.534311 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.534160    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.534311 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.534168    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.534441 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.534420    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.534636 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.534614    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f15b3ac1-60d8-4b82-b686-9e52b6d23868-config-out\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.534699 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.534681    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.534731 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.534709    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f15b3ac1-60d8-4b82-b686-9e52b6d23868-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.536007 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.535984    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f15b3ac1-60d8-4b82-b686-9e52b6d23868-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.541019 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.540996    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2xw\" (UniqueName: \"kubernetes.io/projected/f15b3ac1-60d8-4b82-b686-9e52b6d23868-kube-api-access-fm2xw\") pod \"alertmanager-main-0\" (UID: \"f15b3ac1-60d8-4b82-b686-9e52b6d23868\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.630716 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.630678    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:56:14.770091 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:14.770063    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:56:14.772089 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:56:14.772064    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15b3ac1_60d8_4b82_b686_9e52b6d23868.slice/crio-f39b771647168981d2105a3c5fa52ccd2d8bc11ab4463702f56172e0fc3fcbcd WatchSource:0}: Error finding container f39b771647168981d2105a3c5fa52ccd2d8bc11ab4463702f56172e0fc3fcbcd: Status 404 returned error can't find the container with id f39b771647168981d2105a3c5fa52ccd2d8bc11ab4463702f56172e0fc3fcbcd
Jul 28 12:56:15.253355 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:15.253318    2553 generic.go:356] "Generic (PLEG): container finished" podID="f15b3ac1-60d8-4b82-b686-9e52b6d23868" containerID="808ec33000696ec8b7631d9bb19cacf7f8a6e1e29dfc6f8536c7691d32952c97" exitCode=0
Jul 28 12:56:15.253782 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:15.253378    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f15b3ac1-60d8-4b82-b686-9e52b6d23868","Type":"ContainerDied","Data":"808ec33000696ec8b7631d9bb19cacf7f8a6e1e29dfc6f8536c7691d32952c97"}
Jul 28 12:56:15.253782 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:15.253433    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f15b3ac1-60d8-4b82-b686-9e52b6d23868","Type":"ContainerStarted","Data":"f39b771647168981d2105a3c5fa52ccd2d8bc11ab4463702f56172e0fc3fcbcd"}
Jul 28 12:56:16.040694 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.040658    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"]
Jul 28 12:56:16.043811 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.043770    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.048287 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.047060    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 12:56:16.048287 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.047301    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 12:56:16.048287 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.047544    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 12:56:16.048287 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.047728    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-w4t2f\""
Jul 28 12:56:16.048287 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.047924    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 12:56:16.048287 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.048115    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 12:56:16.052400 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.052380    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 12:56:16.059715 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.059688    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"]
Jul 28 12:56:16.145616 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.145580    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-telemeter-client-tls\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.145616 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.145622    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.145857 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.145658    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgszv\" (UniqueName: \"kubernetes.io/projected/db890d24-1a50-4d69-8909-c7f2d4cb89ef-kube-api-access-sgszv\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.145857 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.145684    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-secret-telemeter-client\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.145857 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.145712    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db890d24-1a50-4d69-8909-c7f2d4cb89ef-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.145857 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.145744    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-federate-client-tls\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.145986 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.145863    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db890d24-1a50-4d69-8909-c7f2d4cb89ef-serving-certs-ca-bundle\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.145986 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.145883    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db890d24-1a50-4d69-8909-c7f2d4cb89ef-metrics-client-ca\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.247166 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.247121    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.247369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.247189    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgszv\" (UniqueName: \"kubernetes.io/projected/db890d24-1a50-4d69-8909-c7f2d4cb89ef-kube-api-access-sgszv\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.247369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.247208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-secret-telemeter-client\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.247369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.247234    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db890d24-1a50-4d69-8909-c7f2d4cb89ef-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.247369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.247261    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-federate-client-tls\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.247369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.247282    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db890d24-1a50-4d69-8909-c7f2d4cb89ef-serving-certs-ca-bundle\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.247369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.247300    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db890d24-1a50-4d69-8909-c7f2d4cb89ef-metrics-client-ca\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.247369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.247322    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-telemeter-client-tls\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.248298 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.248272    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db890d24-1a50-4d69-8909-c7f2d4cb89ef-serving-certs-ca-bundle\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.248478 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.248303    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db890d24-1a50-4d69-8909-c7f2d4cb89ef-metrics-client-ca\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.248612 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.248403    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db890d24-1a50-4d69-8909-c7f2d4cb89ef-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.250074 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.250049    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.250186 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.250154    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-telemeter-client-tls\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.250334 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.250311    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-federate-client-tls\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.250751 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.250732    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/db890d24-1a50-4d69-8909-c7f2d4cb89ef-secret-telemeter-client\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.258486 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.258460    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgszv\" (UniqueName: \"kubernetes.io/projected/db890d24-1a50-4d69-8909-c7f2d4cb89ef-kube-api-access-sgszv\") pod \"telemeter-client-67c8bd987c-rp7hj\" (UID: \"db890d24-1a50-4d69-8909-c7f2d4cb89ef\") " pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.262947 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.262920    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f15b3ac1-60d8-4b82-b686-9e52b6d23868","Type":"ContainerStarted","Data":"4f2bebfacddac4a1a5ccb4539c3f3cb578ed660d0db91e2936e2609bed027c1f"}
Jul 28 12:56:16.263074 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.263003    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f15b3ac1-60d8-4b82-b686-9e52b6d23868","Type":"ContainerStarted","Data":"5cd2a43dd9533878a60f23b71899107e729aaaa91d542b33acea2b02df4999ad"}
Jul 28 12:56:16.263074 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.263054    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f15b3ac1-60d8-4b82-b686-9e52b6d23868","Type":"ContainerStarted","Data":"543abaca5f353ee37c77b194637b22b2dc009bbd2c9a8c0a4bead32882d03696"}
Jul 28 12:56:16.263208 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.263095    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f15b3ac1-60d8-4b82-b686-9e52b6d23868","Type":"ContainerStarted","Data":"a1ca390b93e6c9217d1daef6b97b66c43aab207b5132d23ea5394ed828a7d7ed"}
Jul 28 12:56:16.263208 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.263146    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f15b3ac1-60d8-4b82-b686-9e52b6d23868","Type":"ContainerStarted","Data":"70cd5db50d4f1e505d10b1584e8b4df64acd875a062fb8c732cf91d2796de608"}
Jul 28 12:56:16.263208 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.263186    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f15b3ac1-60d8-4b82-b686-9e52b6d23868","Type":"ContainerStarted","Data":"f3db6ea81d0ca08e86a9584260d177be759ad6defcb0c4327ba03a24e9d32086"}
Jul 28 12:56:16.294611 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.294503    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.278397098 podStartE2EDuration="2.294484146s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.087048ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:56:16.292774011 +0000 UTC m=+146.330142921" watchObservedRunningTime="2026-07-28 12:56:16.294484146 +0000 UTC m=+146.331853054"
Jul 28 12:56:16.356702 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.356658    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"
Jul 28 12:56:16.375402 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.375340    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:56:16.375860 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.375747    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="prometheus" containerID="cri-o://fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e" gracePeriod=600
Jul 28 12:56:16.375986 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.375967    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="thanos-sidecar" containerID="cri-o://959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec" gracePeriod=600
Jul 28 12:56:16.376073 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.375987    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="config-reloader" containerID="cri-o://737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5" gracePeriod=600
Jul 28 12:56:16.376161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.376070    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy" containerID="cri-o://9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52" gracePeriod=600
Jul 28 12:56:16.376161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.376151    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy-web" containerID="cri-o://5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2" gracePeriod=600
Jul 28 12:56:16.376408 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.375964    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd" gracePeriod=600
Jul 28 12:56:16.556496 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.556470    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67c8bd987c-rp7hj"]
Jul 28 12:56:16.558683 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:56:16.558658    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb890d24_1a50_4d69_8909_c7f2d4cb89ef.slice/crio-e8ead981e8f58f67a4ae87e11fbc1746b7ee9946ea750fcca84c7c5836d89f3d WatchSource:0}: Error finding container e8ead981e8f58f67a4ae87e11fbc1746b7ee9946ea750fcca84c7c5836d89f3d: Status 404 returned error can't find the container with id e8ead981e8f58f67a4ae87e11fbc1746b7ee9946ea750fcca84c7c5836d89f3d
Jul 28 12:56:16.654710 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.654679    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:16.755668 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755634    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-grpc-tls\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755668 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755676    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-config-out\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755695    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-kubelet-serving-ca-bundle\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755714    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-trusted-ca-bundle\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755734    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-metrics-client-certs\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755766    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-0\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755803    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-serving-certs-ca-bundle\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755833    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-metrics-client-ca\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755858    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-1\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755891    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-db\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.755952 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755928    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.756406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.755958    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-thanos-prometheus-http-client-file\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.756406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756019    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-tls-assets\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.756406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756056    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-web-config\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.756406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756079    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smlv4\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-kube-api-access-smlv4\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.756406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756122    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.756406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756150    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-kube-rbac-proxy\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.756406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756194    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-2\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.756406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756220    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-config\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.756406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756256    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-tls\") pod \"8842a410-dfa0-4d1d-946e-26fd2a251bee\" (UID: \"8842a410-dfa0-4d1d-946e-26fd2a251bee\") "
Jul 28 12:56:16.758454 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756119    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.758454 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756167    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.758454 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.756366    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.758454 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.757069    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:56:16.758454 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.757393    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.758454 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.757401    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.758454 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.757609    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.758929 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.758740    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.758929 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.758814    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.759199 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.759175    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.762386 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.759456    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-kube-api-access-smlv4" (OuterVolumeSpecName: "kube-api-access-smlv4") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "kube-api-access-smlv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:56:16.762386 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.759603    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-config-out" (OuterVolumeSpecName: "config-out") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:56:16.762386 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.759683    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:56:16.762386 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.759949    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.762386 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.761201    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.762386 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.761475    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.762386 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.762113    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-config" (OuterVolumeSpecName: "config") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.762786 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.762766    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.763263 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.763237    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.773848 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.773821    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-web-config" (OuterVolumeSpecName: "web-config") pod "8842a410-dfa0-4d1d-946e-26fd2a251bee" (UID: "8842a410-dfa0-4d1d-946e-26fd2a251bee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.857153 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857070    2553 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-grpc-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857153 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857100    2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-config-out\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857153 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857111    2553 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857153 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857120    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857153 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857129    2553 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-metrics-client-certs\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857153 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857139    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857153 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857147    2553 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857153 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857155    2553 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-configmap-metrics-client-ca\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857164    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857174    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-db\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857185    2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857194    2553 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857204    2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-tls-assets\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857211    2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-web-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857218    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smlv4\" (UniqueName: \"kubernetes.io/projected/8842a410-dfa0-4d1d-946e-26fd2a251bee-kube-api-access-smlv4\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857226    2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857234    2553 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-kube-rbac-proxy\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857243    2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8842a410-dfa0-4d1d-946e-26fd2a251bee-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857251    2553 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.857511 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:16.857259    2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8842a410-dfa0-4d1d-946e-26fd2a251bee-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 12:56:17.267880 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.267849    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj" event={"ID":"db890d24-1a50-4d69-8909-c7f2d4cb89ef","Type":"ContainerStarted","Data":"e8ead981e8f58f67a4ae87e11fbc1746b7ee9946ea750fcca84c7c5836d89f3d"}
Jul 28 12:56:17.270931 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.270906    2553 generic.go:356] "Generic (PLEG): container finished" podID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerID="ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd" exitCode=0
Jul 28 12:56:17.270931 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.270928    2553 generic.go:356] "Generic (PLEG): container finished" podID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerID="9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52" exitCode=0
Jul 28 12:56:17.270931 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.270935    2553 generic.go:356] "Generic (PLEG): container finished" podID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerID="5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2" exitCode=0
Jul 28 12:56:17.271161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.270941    2553 generic.go:356] "Generic (PLEG): container finished" podID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerID="959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec" exitCode=0
Jul 28 12:56:17.271161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.270947    2553 generic.go:356] "Generic (PLEG): container finished" podID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerID="737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5" exitCode=0
Jul 28 12:56:17.271161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.270952    2553 generic.go:356] "Generic (PLEG): container finished" podID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerID="fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e" exitCode=0
Jul 28 12:56:17.271161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.270982    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.271161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.270978    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerDied","Data":"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"}
Jul 28 12:56:17.271161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.271108    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerDied","Data":"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"}
Jul 28 12:56:17.271161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.271136    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerDied","Data":"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"}
Jul 28 12:56:17.271161 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.271163    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerDied","Data":"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"}
Jul 28 12:56:17.271455 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.271187    2553 scope.go:117] "RemoveContainer" containerID="ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"
Jul 28 12:56:17.271455 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.271187    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerDied","Data":"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"}
Jul 28 12:56:17.271455 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.271291    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerDied","Data":"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"}
Jul 28 12:56:17.271455 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.271324    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8842a410-dfa0-4d1d-946e-26fd2a251bee","Type":"ContainerDied","Data":"19f9f1d5327c2d9409134e677d7204bc1475f883b9291798f71f19a1350c602b"}
Jul 28 12:56:17.279679 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.279662    2553 scope.go:117] "RemoveContainer" containerID="9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"
Jul 28 12:56:17.286399 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.286383    2553 scope.go:117] "RemoveContainer" containerID="5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"
Jul 28 12:56:17.292673 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.292655    2553 scope.go:117] "RemoveContainer" containerID="959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"
Jul 28 12:56:17.299239 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.299168    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:56:17.300593 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.300573    2553 scope.go:117] "RemoveContainer" containerID="737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"
Jul 28 12:56:17.305095 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.305073    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:56:17.307902 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.307878    2553 scope.go:117] "RemoveContainer" containerID="fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"
Jul 28 12:56:17.330354 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.330331    2553 scope.go:117] "RemoveContainer" containerID="edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"
Jul 28 12:56:17.337738 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.337718    2553 scope.go:117] "RemoveContainer" containerID="ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"
Jul 28 12:56:17.338010 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:17.337991    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": container with ID starting with ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd not found: ID does not exist" containerID="ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"
Jul 28 12:56:17.338084 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.338019    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"} err="failed to get container status \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": rpc error: code = NotFound desc = could not find container \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": container with ID starting with ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd not found: ID does not exist"
Jul 28 12:56:17.338084 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.338040    2553 scope.go:117] "RemoveContainer" containerID="9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"
Jul 28 12:56:17.338304 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:17.338280    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": container with ID starting with 9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52 not found: ID does not exist" containerID="9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"
Jul 28 12:56:17.338369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.338309    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"} err="failed to get container status \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": rpc error: code = NotFound desc = could not find container \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": container with ID starting with 9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52 not found: ID does not exist"
Jul 28 12:56:17.338369 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.338330    2553 scope.go:117] "RemoveContainer" containerID="5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"
Jul 28 12:56:17.338593 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:17.338571    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": container with ID starting with 5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2 not found: ID does not exist" containerID="5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"
Jul 28 12:56:17.338679 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.338603    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"} err="failed to get container status \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": rpc error: code = NotFound desc = could not find container \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": container with ID starting with 5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2 not found: ID does not exist"
Jul 28 12:56:17.338679 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.338650    2553 scope.go:117] "RemoveContainer" containerID="959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"
Jul 28 12:56:17.339172 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:17.339135    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": container with ID starting with 959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec not found: ID does not exist" containerID="959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"
Jul 28 12:56:17.339172 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.339162    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"} err="failed to get container status \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": rpc error: code = NotFound desc = could not find container \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": container with ID starting with 959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec not found: ID does not exist"
Jul 28 12:56:17.339378 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.339179    2553 scope.go:117] "RemoveContainer" containerID="737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"
Jul 28 12:56:17.339486 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:17.339468    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": container with ID starting with 737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5 not found: ID does not exist" containerID="737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"
Jul 28 12:56:17.339529 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.339486    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"} err="failed to get container status \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": rpc error: code = NotFound desc = could not find container \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": container with ID starting with 737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5 not found: ID does not exist"
Jul 28 12:56:17.339529 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.339500    2553 scope.go:117] "RemoveContainer" containerID="fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"
Jul 28 12:56:17.339783 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:17.339762    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": container with ID starting with fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e not found: ID does not exist" containerID="fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"
Jul 28 12:56:17.339897 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.339808    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"} err="failed to get container status \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": rpc error: code = NotFound desc = could not find container \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": container with ID starting with fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e not found: ID does not exist"
Jul 28 12:56:17.339897 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.339830    2553 scope.go:117] "RemoveContainer" containerID="edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"
Jul 28 12:56:17.340144 ip-10-0-135-149 kubenswrapper[2553]: E0728 12:56:17.340126    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": container with ID starting with edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d not found: ID does not exist" containerID="edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"
Jul 28 12:56:17.340225 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.340151    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"} err="failed to get container status \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": rpc error: code = NotFound desc = could not find container \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": container with ID starting with edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d not found: ID does not exist"
Jul 28 12:56:17.340225 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.340171    2553 scope.go:117] "RemoveContainer" containerID="ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"
Jul 28 12:56:17.340478 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.340457    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"} err="failed to get container status \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": rpc error: code = NotFound desc = could not find container \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": container with ID starting with ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd not found: ID does not exist"
Jul 28 12:56:17.340478 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.340475    2553 scope.go:117] "RemoveContainer" containerID="9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"
Jul 28 12:56:17.340724 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.340701    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"} err="failed to get container status \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": rpc error: code = NotFound desc = could not find container \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": container with ID starting with 9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52 not found: ID does not exist"
Jul 28 12:56:17.340771 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.340728    2553 scope.go:117] "RemoveContainer" containerID="5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"
Jul 28 12:56:17.340827 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.340776    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:56:17.341007 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.340989    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"} err="failed to get container status \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": rpc error: code = NotFound desc = could not find container \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": container with ID starting with 5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2 not found: ID does not exist"
Jul 28 12:56:17.341046 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341008    2553 scope.go:117] "RemoveContainer" containerID="959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"
Jul 28 12:56:17.341183 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341167    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="prometheus"
Jul 28 12:56:17.341233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341186    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="prometheus"
Jul 28 12:56:17.341233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341196    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="config-reloader"
Jul 28 12:56:17.341233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341202    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="config-reloader"
Jul 28 12:56:17.341233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341215    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy-web"
Jul 28 12:56:17.341233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341221    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy-web"
Jul 28 12:56:17.341233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341220    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"} err="failed to get container status \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": rpc error: code = NotFound desc = could not find container \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": container with ID starting with 959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec not found: ID does not exist"
Jul 28 12:56:17.341233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341228    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy"
Jul 28 12:56:17.341233 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341235    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341246    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy-thanos"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341252    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy-thanos"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341258    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="init-config-reloader"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341262    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="init-config-reloader"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341233    2553 scope.go:117] "RemoveContainer" containerID="737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341270    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="thanos-sidecar"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341350    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="thanos-sidecar"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341445    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="thanos-sidecar"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341459    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341468    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="config-reloader"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341478    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy-thanos"
Jul 28 12:56:17.341493 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341488    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="kube-rbac-proxy-web"
Jul 28 12:56:17.341886 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341499    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" containerName="prometheus"
Jul 28 12:56:17.341886 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341536    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"} err="failed to get container status \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": rpc error: code = NotFound desc = could not find container \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": container with ID starting with 737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5 not found: ID does not exist"
Jul 28 12:56:17.341886 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341551    2553 scope.go:117] "RemoveContainer" containerID="fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"
Jul 28 12:56:17.341886 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341758    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"} err="failed to get container status \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": rpc error: code = NotFound desc = could not find container \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": container with ID starting with fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e not found: ID does not exist"
Jul 28 12:56:17.341886 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.341775    2553 scope.go:117] "RemoveContainer" containerID="edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"
Jul 28 12:56:17.342121 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342020    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"} err="failed to get container status \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": rpc error: code = NotFound desc = could not find container \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": container with ID starting with edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d not found: ID does not exist"
Jul 28 12:56:17.342121 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342042    2553 scope.go:117] "RemoveContainer" containerID="ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"
Jul 28 12:56:17.342280 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342263    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"} err="failed to get container status \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": rpc error: code = NotFound desc = could not find container \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": container with ID starting with ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd not found: ID does not exist"
Jul 28 12:56:17.342323 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342281    2553 scope.go:117] "RemoveContainer" containerID="9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"
Jul 28 12:56:17.342505 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342481    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"} err="failed to get container status \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": rpc error: code = NotFound desc = could not find container \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": container with ID starting with 9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52 not found: ID does not exist"
Jul 28 12:56:17.342505 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342504    2553 scope.go:117] "RemoveContainer" containerID="5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"
Jul 28 12:56:17.342714 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342697    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"} err="failed to get container status \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": rpc error: code = NotFound desc = could not find container \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": container with ID starting with 5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2 not found: ID does not exist"
Jul 28 12:56:17.342714 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342713    2553 scope.go:117] "RemoveContainer" containerID="959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"
Jul 28 12:56:17.342926 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342911    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"} err="failed to get container status \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": rpc error: code = NotFound desc = could not find container \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": container with ID starting with 959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec not found: ID does not exist"
Jul 28 12:56:17.342926 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.342925    2553 scope.go:117] "RemoveContainer" containerID="737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"
Jul 28 12:56:17.343172 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.343145    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"} err="failed to get container status \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": rpc error: code = NotFound desc = could not find container \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": container with ID starting with 737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5 not found: ID does not exist"
Jul 28 12:56:17.343172 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.343166    2553 scope.go:117] "RemoveContainer" containerID="fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"
Jul 28 12:56:17.343378 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.343361    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"} err="failed to get container status \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": rpc error: code = NotFound desc = could not find container \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": container with ID starting with fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e not found: ID does not exist"
Jul 28 12:56:17.343378 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.343377    2553 scope.go:117] "RemoveContainer" containerID="edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"
Jul 28 12:56:17.343586 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.343567    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"} err="failed to get container status \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": rpc error: code = NotFound desc = could not find container \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": container with ID starting with edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d not found: ID does not exist"
Jul 28 12:56:17.343650 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.343586    2553 scope.go:117] "RemoveContainer" containerID="ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"
Jul 28 12:56:17.343841 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.343823    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"} err="failed to get container status \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": rpc error: code = NotFound desc = could not find container \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": container with ID starting with ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd not found: ID does not exist"
Jul 28 12:56:17.343928 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.343842    2553 scope.go:117] "RemoveContainer" containerID="9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"
Jul 28 12:56:17.344051 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344031    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"} err="failed to get container status \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": rpc error: code = NotFound desc = could not find container \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": container with ID starting with 9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52 not found: ID does not exist"
Jul 28 12:56:17.344113 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344053    2553 scope.go:117] "RemoveContainer" containerID="5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"
Jul 28 12:56:17.344276 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344256    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"} err="failed to get container status \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": rpc error: code = NotFound desc = could not find container \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": container with ID starting with 5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2 not found: ID does not exist"
Jul 28 12:56:17.344326 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344276    2553 scope.go:117] "RemoveContainer" containerID="959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"
Jul 28 12:56:17.344458 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344439    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"} err="failed to get container status \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": rpc error: code = NotFound desc = could not find container \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": container with ID starting with 959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec not found: ID does not exist"
Jul 28 12:56:17.344516 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344459    2553 scope.go:117] "RemoveContainer" containerID="737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"
Jul 28 12:56:17.344696 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344678    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"} err="failed to get container status \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": rpc error: code = NotFound desc = could not find container \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": container with ID starting with 737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5 not found: ID does not exist"
Jul 28 12:56:17.344739 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344697    2553 scope.go:117] "RemoveContainer" containerID="fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"
Jul 28 12:56:17.344957 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344936    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"} err="failed to get container status \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": rpc error: code = NotFound desc = could not find container \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": container with ID starting with fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e not found: ID does not exist"
Jul 28 12:56:17.345019 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.344958    2553 scope.go:117] "RemoveContainer" containerID="edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"
Jul 28 12:56:17.345158 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.345143    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.345203 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.345166    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"} err="failed to get container status \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": rpc error: code = NotFound desc = could not find container \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": container with ID starting with edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d not found: ID does not exist"
Jul 28 12:56:17.345203 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.345181    2553 scope.go:117] "RemoveContainer" containerID="ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"
Jul 28 12:56:17.345409 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.345390    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"} err="failed to get container status \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": rpc error: code = NotFound desc = could not find container \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": container with ID starting with ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd not found: ID does not exist"
Jul 28 12:56:17.345454 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.345410    2553 scope.go:117] "RemoveContainer" containerID="9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"
Jul 28 12:56:17.345638 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.345610    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"} err="failed to get container status \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": rpc error: code = NotFound desc = could not find container \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": container with ID starting with 9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52 not found: ID does not exist"
Jul 28 12:56:17.345638 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.345631    2553 scope.go:117] "RemoveContainer" containerID="5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"
Jul 28 12:56:17.345916 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.345896    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"} err="failed to get container status \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": rpc error: code = NotFound desc = could not find container \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": container with ID starting with 5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2 not found: ID does not exist"
Jul 28 12:56:17.345916 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.345916    2553 scope.go:117] "RemoveContainer" containerID="959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"
Jul 28 12:56:17.346154 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.346129    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"} err="failed to get container status \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": rpc error: code = NotFound desc = could not find container \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": container with ID starting with 959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec not found: ID does not exist"
Jul 28 12:56:17.346218 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.346155    2553 scope.go:117] "RemoveContainer" containerID="737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"
Jul 28 12:56:17.346397 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.346380    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"} err="failed to get container status \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": rpc error: code = NotFound desc = could not find container \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": container with ID starting with 737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5 not found: ID does not exist"
Jul 28 12:56:17.346468 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.346398    2553 scope.go:117] "RemoveContainer" containerID="fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"
Jul 28 12:56:17.346610 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.346592    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"} err="failed to get container status \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": rpc error: code = NotFound desc = could not find container \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": container with ID starting with fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e not found: ID does not exist"
Jul 28 12:56:17.346671 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.346610    2553 scope.go:117] "RemoveContainer" containerID="edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"
Jul 28 12:56:17.346876 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.346856    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"} err="failed to get container status \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": rpc error: code = NotFound desc = could not find container \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": container with ID starting with edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d not found: ID does not exist"
Jul 28 12:56:17.346938 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.346876    2553 scope.go:117] "RemoveContainer" containerID="ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"
Jul 28 12:56:17.347162 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.347140    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd"} err="failed to get container status \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": rpc error: code = NotFound desc = could not find container \"ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd\": container with ID starting with ff244c4b70f34aed10110315b7fca1f55a310e90a9d032d055a8d41f5ca9e8bd not found: ID does not exist"
Jul 28 12:56:17.347162 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.347163    2553 scope.go:117] "RemoveContainer" containerID="9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"
Jul 28 12:56:17.347481 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.347453    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52"} err="failed to get container status \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": rpc error: code = NotFound desc = could not find container \"9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52\": container with ID starting with 9387b4e6371cd98e2470d69708b635a9497d6002fde3bda437edf96552ba9b52 not found: ID does not exist"
Jul 28 12:56:17.347570 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.347483    2553 scope.go:117] "RemoveContainer" containerID="5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"
Jul 28 12:56:17.347942 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.347853    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2"} err="failed to get container status \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": rpc error: code = NotFound desc = could not find container \"5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2\": container with ID starting with 5d8f55de6c203bb7449ea2685cc5eecbbf01e4f91155e0003602358084ea6ed2 not found: ID does not exist"
Jul 28 12:56:17.347942 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.347878    2553 scope.go:117] "RemoveContainer" containerID="959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"
Jul 28 12:56:17.351424 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351397    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec"} err="failed to get container status \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": rpc error: code = NotFound desc = could not find container \"959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec\": container with ID starting with 959e411ea8027e1a057de40f04e727eeacc2aae5ac5b26ce27b4cd9ee7a724ec not found: ID does not exist"
Jul 28 12:56:17.351424 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351424    2553 scope.go:117] "RemoveContainer" containerID="737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"
Jul 28 12:56:17.351711 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351687    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 12:56:17.351810 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351703    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5"} err="failed to get container status \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": rpc error: code = NotFound desc = could not find container \"737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5\": container with ID starting with 737f08a450ef49fb81d1954b364b2e801cfc60acdb1d879da878f73be3dc6ac5 not found: ID does not exist"
Jul 28 12:56:17.351810 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351725    2553 scope.go:117] "RemoveContainer" containerID="fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"
Jul 28 12:56:17.351810 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351752    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 12:56:17.352148 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351837    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 12:56:17.352148 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351839    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 12:56:17.352148 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351931    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fuor46sueeiqf\""
Jul 28 12:56:17.352148 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351941    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 12:56:17.352148 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.351896    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 12:56:17.352509 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.352225    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e"} err="failed to get container status \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": rpc error: code = NotFound desc = could not find container \"fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e\": container with ID starting with fb6449919b97e43e0516d63b13361ffd6a933a6b5e0e75ab13069b53d19f794e not found: ID does not exist"
Jul 28 12:56:17.352509 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.352244    2553 scope.go:117] "RemoveContainer" containerID="edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"
Jul 28 12:56:17.352509 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.352253    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 12:56:17.352715 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.352589    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d"} err="failed to get container status \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": rpc error: code = NotFound desc = could not find container \"edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d\": container with ID starting with edc1310adc041be1c982463973e6e309218b4aaed200f87135f2c318f6520f4d not found: ID does not exist"
Jul 28 12:56:17.352773 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.352728    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 12:56:17.352919 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.352728    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 12:56:17.354440 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.353160    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 12:56:17.354440 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.353226    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 12:56:17.357152 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.357111    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 12:56:17.360229 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.360211    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f29cs\""
Jul 28 12:56:17.360327 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.360267    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 12:56:17.361039 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.361018    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 12:56:17.390620 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.390585    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:56:17.462398 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462355    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462398 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462397    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462647 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462424    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462647 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462447    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462647 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462478    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-config\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462647 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462501    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462647 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462522    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462647 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462556    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462906 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462650    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462906 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462682    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fe2553f-bf89-4968-9a57-aa3629efa2d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462906 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462705    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462906 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462766    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462906 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462843    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462906 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462865    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fe2553f-bf89-4968-9a57-aa3629efa2d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462906 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462883    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.462906 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462905    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.463154 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.462987    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.463154 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.463015    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.463154 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.463037    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.463154 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.463051    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4ff4\" (UniqueName: \"kubernetes.io/projected/0fe2553f-bf89-4968-9a57-aa3629efa2d2-kube-api-access-z4ff4\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.563745 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.563673    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.563745 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.563717    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.563745 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.563736    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564022 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.563910    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4ff4\" (UniqueName: \"kubernetes.io/projected/0fe2553f-bf89-4968-9a57-aa3629efa2d2-kube-api-access-z4ff4\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564022 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.563984    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564022 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564008    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564183 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564043    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564183 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564066    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564183 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564105    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-config\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564183 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564143    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564183 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564165    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564199    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564237    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564269    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fe2553f-bf89-4968-9a57-aa3629efa2d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564303    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564344    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564367    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564375    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564405    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fe2553f-bf89-4968-9a57-aa3629efa2d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564480 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564445    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564996 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564485    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.564996 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.564763    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.565190 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.565162    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.565430 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.565364    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.565534 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.565466    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.566229 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.566196    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.567407 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.567076    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.567407 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.567265    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.568658 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.568613    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.569223 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.569171    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.569323 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.569300    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fe2553f-bf89-4968-9a57-aa3629efa2d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.569695 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.569395    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.570201 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.570152    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fe2553f-bf89-4968-9a57-aa3629efa2d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.570576 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.570492    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-config\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.570576 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.570498    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.570874 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.570825    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fe2553f-bf89-4968-9a57-aa3629efa2d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.570989 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.570909    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.571406 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.571382    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.572445 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.572424    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fe2553f-bf89-4968-9a57-aa3629efa2d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.573305 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.573267    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4ff4\" (UniqueName: \"kubernetes.io/projected/0fe2553f-bf89-4968-9a57-aa3629efa2d2-kube-api-access-z4ff4\") pod \"prometheus-k8s-0\" (UID: \"0fe2553f-bf89-4968-9a57-aa3629efa2d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.660288 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.660249    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:17.823230 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:17.823042    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:56:17.825753 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:56:17.825723    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe2553f_bf89_4968_9a57_aa3629efa2d2.slice/crio-5a1ec235349efc06ad77210d84fafe5989b2205c965b46999ab80bcf7680239c WatchSource:0}: Error finding container 5a1ec235349efc06ad77210d84fafe5989b2205c965b46999ab80bcf7680239c: Status 404 returned error can't find the container with id 5a1ec235349efc06ad77210d84fafe5989b2205c965b46999ab80bcf7680239c
Jul 28 12:56:18.278019 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:18.277986    2553 generic.go:356] "Generic (PLEG): container finished" podID="0fe2553f-bf89-4968-9a57-aa3629efa2d2" containerID="1f812c38db195a0be3eae888a9f55dcc928d39477875e4f3bb99948fa1514610" exitCode=0
Jul 28 12:56:18.278484 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:18.278074    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe2553f-bf89-4968-9a57-aa3629efa2d2","Type":"ContainerDied","Data":"1f812c38db195a0be3eae888a9f55dcc928d39477875e4f3bb99948fa1514610"}
Jul 28 12:56:18.278484 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:18.278165    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe2553f-bf89-4968-9a57-aa3629efa2d2","Type":"ContainerStarted","Data":"5a1ec235349efc06ad77210d84fafe5989b2205c965b46999ab80bcf7680239c"}
Jul 28 12:56:18.477984 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:18.477944    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8842a410-dfa0-4d1d-946e-26fd2a251bee" path="/var/lib/kubelet/pods/8842a410-dfa0-4d1d-946e-26fd2a251bee/volumes"
Jul 28 12:56:19.285624 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.285527    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj" event={"ID":"db890d24-1a50-4d69-8909-c7f2d4cb89ef","Type":"ContainerStarted","Data":"0f8ea3d1f56fd71a3a5b51fe2140179a61366cc1ea33bb28464fb85507340b38"}
Jul 28 12:56:19.286052 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.285637    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj" event={"ID":"db890d24-1a50-4d69-8909-c7f2d4cb89ef","Type":"ContainerStarted","Data":"c3df39ba1d4eb00bc3e2894a97e3e48aa6526c3f2bda5a43e1f0d533eba4fb16"}
Jul 28 12:56:19.286052 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.285675    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj" event={"ID":"db890d24-1a50-4d69-8909-c7f2d4cb89ef","Type":"ContainerStarted","Data":"e2cd158e269c97f7ae5000265b8abaffce6e9a5703b75e08c345d1390d98c146"}
Jul 28 12:56:19.288901 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.288875    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe2553f-bf89-4968-9a57-aa3629efa2d2","Type":"ContainerStarted","Data":"d1e41bfaa7bba3631bcb23b04c61435963dfc44fafe04667d3b145d18c1c0918"}
Jul 28 12:56:19.289015 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.288932    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe2553f-bf89-4968-9a57-aa3629efa2d2","Type":"ContainerStarted","Data":"b7a7c5d327d4f4e08766dbf08eab5a955ca9b90b6f362546c162c1dc41b717bf"}
Jul 28 12:56:19.289015 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.288969    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe2553f-bf89-4968-9a57-aa3629efa2d2","Type":"ContainerStarted","Data":"5a89cb92bf328e805fa5dbbfbdcda0070ec7ccd2f9b424b832f72f593d2e9739"}
Jul 28 12:56:19.289015 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.288994    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe2553f-bf89-4968-9a57-aa3629efa2d2","Type":"ContainerStarted","Data":"48336f2506f355cc3b3409f66cef704a8b703041b65fbd5f7ef4669d3f205811"}
Jul 28 12:56:19.289125 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.289017    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe2553f-bf89-4968-9a57-aa3629efa2d2","Type":"ContainerStarted","Data":"33cbc8371b07f9ca837e19f0df29eb0f59025c003ad3d4377a08e1565e30adc8"}
Jul 28 12:56:19.289125 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.289041    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe2553f-bf89-4968-9a57-aa3629efa2d2","Type":"ContainerStarted","Data":"a7d753f8699d7caa46b09ad989d5644068130a3e6d2f8147535b32d24e14c83f"}
Jul 28 12:56:19.314887 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.314832    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-67c8bd987c-rp7hj" podStartSLOduration=1.190135218 podStartE2EDuration="3.314812512s" totalImagesPullingTime="2.124677294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:56:19.309105814 +0000 UTC m=+149.346474722" watchObservedRunningTime="2026-07-28 12:56:19.314812512 +0000 UTC m=+149.352181415"
Jul 28 12:56:19.343890 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:19.343826    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.32192904 podStartE2EDuration="2.3438071s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.87806ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:56:19.342147762 +0000 UTC m=+149.379516670" watchObservedRunningTime="2026-07-28 12:56:19.3438071 +0000 UTC m=+149.381175998"
Jul 28 12:56:20.295378 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:20.295342    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:20.421216 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:20.421171    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:21.023876 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:21.023837    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:21.090702 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:21.090661    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:22.660711 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:22.660679    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:56:31.725896 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:31.725849    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:31.828170 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:31.828141    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:31.927071 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:31.927038    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:41.765202 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:41.765168    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:47.592248 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:47.592206    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:48.401339 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:48.401301    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:56:55.742145 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:56:55.742105    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:17.660441 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:17.660403    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:57:17.676697 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:17.676673    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:57:18.534347 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:18.534318    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:57:26.205243 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:26.205198    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:26.266271 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:26.266241    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:35.286617 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:35.286577    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:35.446532 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:35.446498    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:35.506413 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:35.506383    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:35.990283 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:35.990247    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:36.042998 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:36.042973    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:38.251180 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:38.251146    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:38.318669 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:38.318636    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:45.971267 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:45.971226    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:46.072112 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:46.072084    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:46.136063 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:46.136031    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:48.208477 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:48.208442    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:48.309327 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:48.309294    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:55.836096 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:55.836054    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:57:55.898697 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:57:55.898666    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:58:50.309929 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:58:50.309892    2553 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 12:59:11.998840 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:11.998806    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:13.289300 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:13.289270    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:39.891195 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:39.891160    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:39.986085 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:39.986057    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:45.805087 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:45.804994    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:46.099832 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:46.099731    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:46.375565 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:46.375470    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:46.981178 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:46.981135    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:47.166933 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.166901    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-njtcw"]
Jul 28 12:59:47.170246 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.170226    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 12:59:47.182429 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.182401    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-l6c5l\""
Jul 28 12:59:47.182693 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.182675    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 12:59:47.182895 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.182878    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 12:59:47.183648 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.183624    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 12:59:47.183730 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.183651    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 12:59:47.186075 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.186054    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:47.188479 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.188460    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-njtcw"]
Jul 28 12:59:47.288757 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.288664    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0d251298-0ad5-4825-a2ff-77f5b5f32248-certificates\") pod \"keda-admission-56f9798bb4-njtcw\" (UID: \"0d251298-0ad5-4825-a2ff-77f5b5f32248\") " pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 12:59:47.288757 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.288747    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4n5w\" (UniqueName: \"kubernetes.io/projected/0d251298-0ad5-4825-a2ff-77f5b5f32248-kube-api-access-b4n5w\") pod \"keda-admission-56f9798bb4-njtcw\" (UID: \"0d251298-0ad5-4825-a2ff-77f5b5f32248\") " pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 12:59:47.389351 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.389313    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4n5w\" (UniqueName: \"kubernetes.io/projected/0d251298-0ad5-4825-a2ff-77f5b5f32248-kube-api-access-b4n5w\") pod \"keda-admission-56f9798bb4-njtcw\" (UID: \"0d251298-0ad5-4825-a2ff-77f5b5f32248\") " pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 12:59:47.389542 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.389378    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0d251298-0ad5-4825-a2ff-77f5b5f32248-certificates\") pod \"keda-admission-56f9798bb4-njtcw\" (UID: \"0d251298-0ad5-4825-a2ff-77f5b5f32248\") " pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 12:59:47.391785 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.391760    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0d251298-0ad5-4825-a2ff-77f5b5f32248-certificates\") pod \"keda-admission-56f9798bb4-njtcw\" (UID: \"0d251298-0ad5-4825-a2ff-77f5b5f32248\") " pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 12:59:47.400870 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.400852    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4n5w\" (UniqueName: \"kubernetes.io/projected/0d251298-0ad5-4825-a2ff-77f5b5f32248-kube-api-access-b4n5w\") pod \"keda-admission-56f9798bb4-njtcw\" (UID: \"0d251298-0ad5-4825-a2ff-77f5b5f32248\") " pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 12:59:47.404418 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.404395    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:47.480755 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.480712    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 12:59:47.611813 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.611770    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:47.646592 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.646560    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-njtcw"]
Jul 28 12:59:47.647910 ip-10-0-135-149 kubenswrapper[2553]: W0728 12:59:47.647878    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d251298_0ad5_4825_a2ff_77f5b5f32248.slice/crio-81d8a83d1a583b931033db319115c647f7dfb9cbf6d3c227925427e857f0e81c WatchSource:0}: Error finding container 81d8a83d1a583b931033db319115c647f7dfb9cbf6d3c227925427e857f0e81c: Status 404 returned error can't find the container with id 81d8a83d1a583b931033db319115c647f7dfb9cbf6d3c227925427e857f0e81c
Jul 28 12:59:47.649273 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.649255    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:59:47.821848 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:47.821744    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:48.023621 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:48.023591    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:48.093131 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:48.093038    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-njtcw" event={"ID":"0d251298-0ad5-4825-a2ff-77f5b5f32248","Type":"ContainerStarted","Data":"81d8a83d1a583b931033db319115c647f7dfb9cbf6d3c227925427e857f0e81c"}
Jul 28 12:59:48.120181 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:48.120142    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 12:59:51.107130 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:51.107098    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-njtcw" event={"ID":"0d251298-0ad5-4825-a2ff-77f5b5f32248","Type":"ContainerStarted","Data":"7efe39bed24fbdf62b83e280960ba3625b21f794c9bb898b966f5dddfa92269f"}
Jul 28 12:59:51.107499 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:51.107213    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 12:59:51.124578 ip-10-0-135-149 kubenswrapper[2553]: I0728 12:59:51.124536    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-njtcw" podStartSLOduration=1.546855936 podStartE2EDuration="4.124522717s" totalImagesPullingTime="2.577666781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:59:51.124120373 +0000 UTC m=+361.161489270" watchObservedRunningTime="2026-07-28 12:59:51.124522717 +0000 UTC m=+361.161891623"
Jul 28 13:00:12.113028 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:00:12.112996    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-njtcw"
Jul 28 13:00:30.441811 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:00:30.441760    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:00:30.499845 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:00:30.499816    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:00:32.314830 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:00:32.314774    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:00:32.368226 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:00:32.368197    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:06.152101 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.152067    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:06.229706 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.229673    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:06.271941 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.271907    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-lpddd"]
Jul 28 13:01:06.275417 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.275396    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:06.278444 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.278422    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xk4xb\""
Jul 28 13:01:06.279339 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.279318    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 13:01:06.279555 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.279539    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 13:01:06.279663 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.279648    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 13:01:06.284878 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.284857    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-lpddd"]
Jul 28 13:01:06.350943 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.350904    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:06.390999 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.390968    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/25cea127-7ab1-41f8-a26b-e90fcf628800-data\") pod \"seaweedfs-64568bcd49-lpddd\" (UID: \"25cea127-7ab1-41f8-a26b-e90fcf628800\") " pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:06.391110 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.391008    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rrfx\" (UniqueName: \"kubernetes.io/projected/25cea127-7ab1-41f8-a26b-e90fcf628800-kube-api-access-5rrfx\") pod \"seaweedfs-64568bcd49-lpddd\" (UID: \"25cea127-7ab1-41f8-a26b-e90fcf628800\") " pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:06.491865 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.491838    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/25cea127-7ab1-41f8-a26b-e90fcf628800-data\") pod \"seaweedfs-64568bcd49-lpddd\" (UID: \"25cea127-7ab1-41f8-a26b-e90fcf628800\") " pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:06.491997 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.491869    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rrfx\" (UniqueName: \"kubernetes.io/projected/25cea127-7ab1-41f8-a26b-e90fcf628800-kube-api-access-5rrfx\") pod \"seaweedfs-64568bcd49-lpddd\" (UID: \"25cea127-7ab1-41f8-a26b-e90fcf628800\") " pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:06.492256 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.492241    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/25cea127-7ab1-41f8-a26b-e90fcf628800-data\") pod \"seaweedfs-64568bcd49-lpddd\" (UID: \"25cea127-7ab1-41f8-a26b-e90fcf628800\") " pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:06.500911 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.500882    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rrfx\" (UniqueName: \"kubernetes.io/projected/25cea127-7ab1-41f8-a26b-e90fcf628800-kube-api-access-5rrfx\") pod \"seaweedfs-64568bcd49-lpddd\" (UID: \"25cea127-7ab1-41f8-a26b-e90fcf628800\") " pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:06.584889 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.584864    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:06.618347 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.618305    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:06.709286 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.709262    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-lpddd"]
Jul 28 13:01:06.712080 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:01:06.712052    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cea127_7ab1_41f8_a26b_e90fcf628800.slice/crio-142f300f599b86dea223406e7c7ebf91651afac5b1d6574554981aafd6095c24 WatchSource:0}: Error finding container 142f300f599b86dea223406e7c7ebf91651afac5b1d6574554981aafd6095c24: Status 404 returned error can't find the container with id 142f300f599b86dea223406e7c7ebf91651afac5b1d6574554981aafd6095c24
Jul 28 13:01:06.712813 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.712761    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:06.870854 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.870753    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:06.969344 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:06.969301    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:07.069276 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:07.069245    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:07.414032 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:07.414002    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-lpddd" event={"ID":"25cea127-7ab1-41f8-a26b-e90fcf628800","Type":"ContainerStarted","Data":"142f300f599b86dea223406e7c7ebf91651afac5b1d6574554981aafd6095c24"}
Jul 28 13:01:10.428916 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:10.428883    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-lpddd" event={"ID":"25cea127-7ab1-41f8-a26b-e90fcf628800","Type":"ContainerStarted","Data":"c19e57ea248517601bb4e6d31cbe71756077384349e971acb0957e7f589176bf"}
Jul 28 13:01:10.429285 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:10.429064    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:10.447313 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:10.447267    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-lpddd" podStartSLOduration=1.138094837 podStartE2EDuration="4.447254534s" totalImagesPullingTime="3.309159697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:01:10.446338262 +0000 UTC m=+440.483707182" watchObservedRunningTime="2026-07-28 13:01:10.447254534 +0000 UTC m=+440.484623441"
Jul 28 13:01:16.436175 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:16.436097    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-lpddd"
Jul 28 13:01:21.274985 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:21.274950    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:21.466264 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:21.466224    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:31.727383 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:31.727346    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:31.829590 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:31.829556    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:31.933111 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:31.933079    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:01:35.734874 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:01:35.734837    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:23.573005 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:23.572969    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:23.661505 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:23.661451    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:23.927239 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:23.927212    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:24.028696 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:24.028662    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:24.338926 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:24.338834    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:24.440786 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:24.440753    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:26.453070 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:26.453035    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:02:41.439746 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.439713    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-fx5b6"]
Jul 28 13:02:41.443407 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.443384    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fx5b6"
Jul 28 13:02:41.451233 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.451204    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-fx5b6"]
Jul 28 13:02:41.464054 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.464027    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7br6\" (UniqueName: \"kubernetes.io/projected/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-kube-api-access-s7br6\") pod \"s3-init-fx5b6\" (UID: \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\") " pod="kserve/s3-init-fx5b6"
Jul 28 13:02:41.464199 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.464076    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-model-cache\") pod \"s3-init-fx5b6\" (UID: \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\") " pod="kserve/s3-init-fx5b6"
Jul 28 13:02:41.565338 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.565300    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-model-cache\") pod \"s3-init-fx5b6\" (UID: \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\") " pod="kserve/s3-init-fx5b6"
Jul 28 13:02:41.565497 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.565393    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7br6\" (UniqueName: \"kubernetes.io/projected/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-kube-api-access-s7br6\") pod \"s3-init-fx5b6\" (UID: \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\") " pod="kserve/s3-init-fx5b6"
Jul 28 13:02:41.565748 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.565730    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-model-cache\") pod \"s3-init-fx5b6\" (UID: \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\") " pod="kserve/s3-init-fx5b6"
Jul 28 13:02:41.575302 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.575275    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7br6\" (UniqueName: \"kubernetes.io/projected/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-kube-api-access-s7br6\") pod \"s3-init-fx5b6\" (UID: \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\") " pod="kserve/s3-init-fx5b6"
Jul 28 13:02:41.761144 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.761067    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fx5b6"
Jul 28 13:02:41.888713 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:41.888685    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-fx5b6"]
Jul 28 13:02:41.890722 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:02:41.890692    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2fcbdd3_610b_4bc7_8c11_e49961937aa0.slice/crio-7c19b5e71a382116888dc500f6274bdb9b50372c6bc69b96d7537c66533024d0 WatchSource:0}: Error finding container 7c19b5e71a382116888dc500f6274bdb9b50372c6bc69b96d7537c66533024d0: Status 404 returned error can't find the container with id 7c19b5e71a382116888dc500f6274bdb9b50372c6bc69b96d7537c66533024d0
Jul 28 13:02:42.796406 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:42.796374    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fx5b6" event={"ID":"c2fcbdd3-610b-4bc7-8c11-e49961937aa0","Type":"ContainerStarted","Data":"7c19b5e71a382116888dc500f6274bdb9b50372c6bc69b96d7537c66533024d0"}
Jul 28 13:02:45.812376 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:45.812351    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fx5b6" event={"ID":"c2fcbdd3-610b-4bc7-8c11-e49961937aa0","Type":"ContainerStarted","Data":"11fe8ab1783a6742648cef58b4dcca200d450dcf5235a0242c0bc0da3ce6eb23"}
Jul 28 13:02:48.826508 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:48.826422    2553 generic.go:356] "Generic (PLEG): container finished" podID="c2fcbdd3-610b-4bc7-8c11-e49961937aa0" containerID="11fe8ab1783a6742648cef58b4dcca200d450dcf5235a0242c0bc0da3ce6eb23" exitCode=0
Jul 28 13:02:48.826508 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:48.826474    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fx5b6" event={"ID":"c2fcbdd3-610b-4bc7-8c11-e49961937aa0","Type":"ContainerDied","Data":"11fe8ab1783a6742648cef58b4dcca200d450dcf5235a0242c0bc0da3ce6eb23"}
Jul 28 13:02:53.852627 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:53.852590    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fx5b6" event={"ID":"c2fcbdd3-610b-4bc7-8c11-e49961937aa0","Type":"ContainerStarted","Data":"bceec8d3b1036c0bc8cb88968ae56c71012e52a06fb1d2c3af14812bda8231b1"}
Jul 28 13:02:53.872317 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:53.872262    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-fx5b6" podStartSLOduration=2.439337611 podStartE2EDuration="12.872245499s" totalImagesPullingTime="8.308369768s" totalInitContainerRuntime="2.12453812s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:02:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:02:53.871585219 +0000 UTC m=+543.908954128" watchObservedRunningTime="2026-07-28 13:02:53.872245499 +0000 UTC m=+543.909614409"
Jul 28 13:02:53.873082 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:02:53.873047    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:03:01.971185 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:03:01.971142    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:03:59.129505 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:03:59.129469    2553 generic.go:356] "Generic (PLEG): container finished" podID="c2fcbdd3-610b-4bc7-8c11-e49961937aa0" containerID="bceec8d3b1036c0bc8cb88968ae56c71012e52a06fb1d2c3af14812bda8231b1" exitCode=0
Jul 28 13:03:59.129943 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:03:59.129521    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fx5b6" event={"ID":"c2fcbdd3-610b-4bc7-8c11-e49961937aa0","Type":"ContainerDied","Data":"bceec8d3b1036c0bc8cb88968ae56c71012e52a06fb1d2c3af14812bda8231b1"}
Jul 28 13:04:00.261060 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:00.261034    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fx5b6"
Jul 28 13:04:00.317052 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:00.317018    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7br6\" (UniqueName: \"kubernetes.io/projected/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-kube-api-access-s7br6\") pod \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\" (UID: \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\") "
Jul 28 13:04:00.317215 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:00.317059    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-model-cache\") pod \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\" (UID: \"c2fcbdd3-610b-4bc7-8c11-e49961937aa0\") "
Jul 28 13:04:00.319293 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:00.319261    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-kube-api-access-s7br6" (OuterVolumeSpecName: "kube-api-access-s7br6") pod "c2fcbdd3-610b-4bc7-8c11-e49961937aa0" (UID: "c2fcbdd3-610b-4bc7-8c11-e49961937aa0"). InnerVolumeSpecName "kube-api-access-s7br6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:04:00.335936 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:00.335907    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2fcbdd3-610b-4bc7-8c11-e49961937aa0" (UID: "c2fcbdd3-610b-4bc7-8c11-e49961937aa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:04:00.418117 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:00.418088    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7br6\" (UniqueName: \"kubernetes.io/projected/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-kube-api-access-s7br6\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:04:00.418117 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:00.418118    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2fcbdd3-610b-4bc7-8c11-e49961937aa0-model-cache\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:04:01.138711 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:01.138676    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-fx5b6"
Jul 28 13:04:01.138711 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:01.138693    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-fx5b6" event={"ID":"c2fcbdd3-610b-4bc7-8c11-e49961937aa0","Type":"ContainerDied","Data":"7c19b5e71a382116888dc500f6274bdb9b50372c6bc69b96d7537c66533024d0"}
Jul 28 13:04:01.138938 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:01.138725    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c19b5e71a382116888dc500f6274bdb9b50372c6bc69b96d7537c66533024d0"
Jul 28 13:04:01.534479 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:01.534442    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:04:01.635568 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:01.635533    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:04:15.128153 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.128118    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:04:15.471617 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.471579    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"]
Jul 28 13:04:15.472016 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.471998    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2fcbdd3-610b-4bc7-8c11-e49961937aa0" containerName="s3-init"
Jul 28 13:04:15.472113 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.472017    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2fcbdd3-610b-4bc7-8c11-e49961937aa0" containerName="s3-init"
Jul 28 13:04:15.472113 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.472035    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2fcbdd3-610b-4bc7-8c11-e49961937aa0" containerName="download-hf-model"
Jul 28 13:04:15.472113 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.472044    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2fcbdd3-610b-4bc7-8c11-e49961937aa0" containerName="download-hf-model"
Jul 28 13:04:15.472269 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.472136    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2fcbdd3-610b-4bc7-8c11-e49961937aa0" containerName="s3-init"
Jul 28 13:04:15.475351 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.475327    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.478452 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.478431    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:04:15.478647 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.478632    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 28 13:04:15.478755 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.478661    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:04:15.478755 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.478706    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:04:15.478755 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.478732    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 28 13:04:15.479041 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.479025    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:04:15.480100 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.480075    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pkltr\""
Jul 28 13:04:15.484970 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.484951    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"]
Jul 28 13:04:15.547188 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.547158    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"]
Jul 28 13:04:15.550495 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.550478    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.553234 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.553214    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-aee94-predictor-serving-cert\""
Jul 28 13:04:15.553433 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.553419    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-aee94-kube-rbac-proxy-sar-config\""
Jul 28 13:04:15.557026 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.557003    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.557280 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.557252    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9npbh\" (UniqueName: \"kubernetes.io/projected/f773907b-7eee-4385-ae85-d2134155cd42-kube-api-access-9npbh\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.557452 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.557426    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.557568 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.557517    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f773907b-7eee-4385-ae85-d2134155cd42-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.557568 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.557556    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f773907b-7eee-4385-ae85-d2134155cd42-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.559557 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.559534    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"]
Jul 28 13:04:15.658096 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658057    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f773907b-7eee-4385-ae85-d2134155cd42-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.658260 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658101    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aee94-predictor-6c8ccbb46b-q678d\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") " pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.658260 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658220    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f773907b-7eee-4385-ae85-d2134155cd42-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.658337 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658258    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.658337 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658315    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-proxy-tls\") pod \"error-404-isvc-aee94-predictor-6c8ccbb46b-q678d\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") " pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.658408 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658346    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9npbh\" (UniqueName: \"kubernetes.io/projected/f773907b-7eee-4385-ae85-d2134155cd42-kube-api-access-9npbh\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.658408 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:04:15.658376    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 28 13:04:15.658516 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:04:15.658447    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f773907b-7eee-4385-ae85-d2134155cd42-proxy-tls podName:f773907b-7eee-4385-ae85-d2134155cd42 nodeName:}" failed. No retries permitted until 2026-07-28 13:04:16.158426896 +0000 UTC m=+626.195795782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f773907b-7eee-4385-ae85-d2134155cd42-proxy-tls") pod "isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" (UID: "f773907b-7eee-4385-ae85-d2134155cd42") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 28 13:04:15.658516 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658459    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f773907b-7eee-4385-ae85-d2134155cd42-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.658516 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658376    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.658650 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658562    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll9nb\" (UniqueName: \"kubernetes.io/projected/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-api-access-ll9nb\") pod \"error-404-isvc-aee94-predictor-6c8ccbb46b-q678d\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") " pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.658932 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.658915    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.659068 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.659052    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.668276 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.668241    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9npbh\" (UniqueName: \"kubernetes.io/projected/f773907b-7eee-4385-ae85-d2134155cd42-kube-api-access-9npbh\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:15.677427 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.677399    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"]
Jul 28 13:04:15.681121 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.681106    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.683926 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.683907    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 28 13:04:15.683926 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.683922    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 28 13:04:15.690612 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.690587    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"]
Jul 28 13:04:15.760058 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.759982    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aee94-predictor-6c8ccbb46b-q678d\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") " pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.760058 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.760051    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d250e58-bb57-4e03-bebe-f22ba62ea946-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.760281 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.760071    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.760281 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.760093    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhgm5\" (UniqueName: \"kubernetes.io/projected/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-api-access-nhgm5\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.760281 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.760125    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d250e58-bb57-4e03-bebe-f22ba62ea946-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.760281 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.760157    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-proxy-tls\") pod \"error-404-isvc-aee94-predictor-6c8ccbb46b-q678d\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") " pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.760281 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.760224    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.760502 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.760316    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9nb\" (UniqueName: \"kubernetes.io/projected/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-api-access-ll9nb\") pod \"error-404-isvc-aee94-predictor-6c8ccbb46b-q678d\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") " pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.760845 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.760823    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aee94-predictor-6c8ccbb46b-q678d\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") " pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.762970 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.762944    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-proxy-tls\") pod \"error-404-isvc-aee94-predictor-6c8ccbb46b-q678d\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") " pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.771389 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.771362    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll9nb\" (UniqueName: \"kubernetes.io/projected/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-api-access-ll9nb\") pod \"error-404-isvc-aee94-predictor-6c8ccbb46b-q678d\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") " pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.861590 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.861552    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:15.861826 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.861778    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d250e58-bb57-4e03-bebe-f22ba62ea946-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.861884 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.861841    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.861884 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.861873    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhgm5\" (UniqueName: \"kubernetes.io/projected/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-api-access-nhgm5\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.861957 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.861908    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d250e58-bb57-4e03-bebe-f22ba62ea946-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.861995 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.861957    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.862424 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.862392    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d250e58-bb57-4e03-bebe-f22ba62ea946-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.862604 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.862585    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.862604 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.862595    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.864348 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.864327    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d250e58-bb57-4e03-bebe-f22ba62ea946-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.873630 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.873608    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhgm5\" (UniqueName: \"kubernetes.io/projected/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-api-access-nhgm5\") pod \"isvc-xgboost-graph-predictor-555fc69d56-98h27\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.992373 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.992338    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:15.996679 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:15.996617    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"]
Jul 28 13:04:15.998717 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:04:15.998690    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd0601b_6fa6_407f_aaf2_b9741e3834c2.slice/crio-ddc89a128a5a0dcb8028edd0655792d8a672f3ab2040b31ce43bbe4dba141d35 WatchSource:0}: Error finding container ddc89a128a5a0dcb8028edd0655792d8a672f3ab2040b31ce43bbe4dba141d35: Status 404 returned error can't find the container with id ddc89a128a5a0dcb8028edd0655792d8a672f3ab2040b31ce43bbe4dba141d35
Jul 28 13:04:16.124511 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:16.124486    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"]
Jul 28 13:04:16.126267 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:04:16.126242    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d250e58_bb57_4e03_bebe_f22ba62ea946.slice/crio-d47d833aee362227c2e1ac04270198f36500721e0db3443e5af7b03679cf821e WatchSource:0}: Error finding container d47d833aee362227c2e1ac04270198f36500721e0db3443e5af7b03679cf821e: Status 404 returned error can't find the container with id d47d833aee362227c2e1ac04270198f36500721e0db3443e5af7b03679cf821e
Jul 28 13:04:16.165846 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:16.165821    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f773907b-7eee-4385-ae85-d2134155cd42-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:16.168212 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:16.168191    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f773907b-7eee-4385-ae85-d2134155cd42-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:16.201685 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:16.201650    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" event={"ID":"2dd0601b-6fa6-407f-aaf2-b9741e3834c2","Type":"ContainerStarted","Data":"ddc89a128a5a0dcb8028edd0655792d8a672f3ab2040b31ce43bbe4dba141d35"}
Jul 28 13:04:16.203118 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:16.203092    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" event={"ID":"5d250e58-bb57-4e03-bebe-f22ba62ea946","Type":"ContainerStarted","Data":"638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb"}
Jul 28 13:04:16.203212 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:16.203151    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" event={"ID":"5d250e58-bb57-4e03-bebe-f22ba62ea946","Type":"ContainerStarted","Data":"d47d833aee362227c2e1ac04270198f36500721e0db3443e5af7b03679cf821e"}
Jul 28 13:04:16.386255 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:16.386171    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:16.521565 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:16.521535    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"]
Jul 28 13:04:16.523963 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:04:16.523923    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf773907b_7eee_4385_ae85_d2134155cd42.slice/crio-157db00c98292f5b51e4dd3dba323795313c59b7d1790579f18b8ea6bc8af27e WatchSource:0}: Error finding container 157db00c98292f5b51e4dd3dba323795313c59b7d1790579f18b8ea6bc8af27e: Status 404 returned error can't find the container with id 157db00c98292f5b51e4dd3dba323795313c59b7d1790579f18b8ea6bc8af27e
Jul 28 13:04:17.214330 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:17.213962    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" event={"ID":"f773907b-7eee-4385-ae85-d2134155cd42","Type":"ContainerStarted","Data":"6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66"}
Jul 28 13:04:17.214330 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:17.214036    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" event={"ID":"f773907b-7eee-4385-ae85-d2134155cd42","Type":"ContainerStarted","Data":"157db00c98292f5b51e4dd3dba323795313c59b7d1790579f18b8ea6bc8af27e"}
Jul 28 13:04:17.581019 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:17.580844    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:04:21.241984 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:21.241949    2553 generic.go:356] "Generic (PLEG): container finished" podID="f773907b-7eee-4385-ae85-d2134155cd42" containerID="6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66" exitCode=0
Jul 28 13:04:21.242457 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:21.242038    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" event={"ID":"f773907b-7eee-4385-ae85-d2134155cd42","Type":"ContainerDied","Data":"6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66"}
Jul 28 13:04:21.243982 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:21.243856    2553 generic.go:356] "Generic (PLEG): container finished" podID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerID="638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb" exitCode=0
Jul 28 13:04:21.243982 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:21.243887    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" event={"ID":"5d250e58-bb57-4e03-bebe-f22ba62ea946","Type":"ContainerDied","Data":"638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb"}
Jul 28 13:04:32.306316 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:32.306246    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" event={"ID":"f773907b-7eee-4385-ae85-d2134155cd42","Type":"ContainerStarted","Data":"3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6"}
Jul 28 13:04:32.308047 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:32.308017    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" event={"ID":"2dd0601b-6fa6-407f-aaf2-b9741e3834c2","Type":"ContainerStarted","Data":"c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8"}
Jul 28 13:04:36.332688 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.332602    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" event={"ID":"2dd0601b-6fa6-407f-aaf2-b9741e3834c2","Type":"ContainerStarted","Data":"8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf"}
Jul 28 13:04:36.333115 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.332826    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:36.333115 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.332891    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:36.334399 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.334365    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 13:04:36.335049 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.335020    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" event={"ID":"f773907b-7eee-4385-ae85-d2134155cd42","Type":"ContainerStarted","Data":"b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97"}
Jul 28 13:04:36.335280 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.335265    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:36.335356 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.335298    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:36.336273 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.336243    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:04:36.352198 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.352160    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podStartSLOduration=1.52454656 podStartE2EDuration="21.352146789s" totalImagesPullingTime="19.827600229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:36.351180519 +0000 UTC m=+646.388549428" watchObservedRunningTime="2026-07-28 13:04:36.352146789 +0000 UTC m=+646.389515696"
Jul 28 13:04:36.372613 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:36.372578    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podStartSLOduration=2.757465931 podStartE2EDuration="21.372566803s" totalImagesPullingTime="14.585328923s" totalInitContainerRuntime="4.029771949s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:36.370866239 +0000 UTC m=+646.408235148" watchObservedRunningTime="2026-07-28 13:04:36.372566803 +0000 UTC m=+646.409935711"
Jul 28 13:04:37.340530 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:37.340487    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 13:04:37.341007 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:37.340627    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:04:42.345871 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:42.345843    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:04:42.346443 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:42.346424    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:04:42.346568 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:42.346534    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:04:42.346896 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:42.346864    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 13:04:52.347146 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:52.347060    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:04:52.347559 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:52.347067    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 13:04:52.415082 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:52.415049    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" event={"ID":"5d250e58-bb57-4e03-bebe-f22ba62ea946","Type":"ContainerStarted","Data":"3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393"}
Jul 28 13:04:52.415242 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:52.415117    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" event={"ID":"5d250e58-bb57-4e03-bebe-f22ba62ea946","Type":"ContainerStarted","Data":"d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223"}
Jul 28 13:04:52.415410 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:52.415391    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:52.415553 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:52.415537    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:52.416673 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:52.416652    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 13:04:52.443584 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:52.443544    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podStartSLOduration=2.672961194 podStartE2EDuration="37.443531096s" totalImagesPullingTime="30.742337803s" totalInitContainerRuntime="4.028232099s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:52.442340779 +0000 UTC m=+662.479709686" watchObservedRunningTime="2026-07-28 13:04:52.443531096 +0000 UTC m=+662.480900003"
Jul 28 13:04:53.419780 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:53.419739    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 13:04:58.424595 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:58.424571    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:04:58.425053 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:04:58.425026    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 13:05:02.346567 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:02.346522    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:05:02.346972 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:02.346851    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 13:05:07.674640 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:07.674608    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:08.425280 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:08.425238    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 13:05:12.346387 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:12.346346    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:05:12.346773 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:12.346747    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 13:05:18.425276 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:18.425235    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 13:05:22.347057 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:22.347010    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:05:22.347586 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:22.347568    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:05:28.425426 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:28.425385    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 13:05:32.346471 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:32.346428    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:05:35.189927 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:35.189862    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:35.892045 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:35.892004    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:38.425967 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:38.425930    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 13:05:42.347025 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:42.346932    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:05:43.471034 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:43.470994    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 13:05:48.425522 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:48.425490    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:05:49.402715 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.402675    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:49.562125 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.562067    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:49.660816 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.660723    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:49.685463 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.685432    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"]
Jul 28 13:05:49.688235 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.688213    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:49.691429 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.691400    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c21f8-predictor-serving-cert\""
Jul 28 13:05:49.691632 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.691607    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c21f8-kube-rbac-proxy-sar-config\""
Jul 28 13:05:49.700862 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.700836    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"]
Jul 28 13:05:49.714816 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.714769    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2pvh\" (UniqueName: \"kubernetes.io/projected/c9980558-0b64-49ae-900a-c16cdc68794e-kube-api-access-l2pvh\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:49.714940 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.714856    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:49.714940 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.714889    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9980558-0b64-49ae-900a-c16cdc68794e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:49.720072 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.720051    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"]
Jul 28 13:05:49.720261 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.720236    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container" containerID="cri-o://c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8" gracePeriod=30
Jul 28 13:05:49.720318 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.720262    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kube-rbac-proxy" containerID="cri-o://8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf" gracePeriod=30
Jul 28 13:05:49.744467 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.744448    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:49.815522 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.815500    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pvh\" (UniqueName: \"kubernetes.io/projected/c9980558-0b64-49ae-900a-c16cdc68794e-kube-api-access-l2pvh\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:49.815625 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.815569    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:49.815625 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.815604    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9980558-0b64-49ae-900a-c16cdc68794e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:49.815729 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:05:49.815675    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-serving-cert: secret "error-404-isvc-c21f8-predictor-serving-cert" not found
Jul 28 13:05:49.815766 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:05:49.815729    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls podName:c9980558-0b64-49ae-900a-c16cdc68794e nodeName:}" failed. No retries permitted until 2026-07-28 13:05:50.315716426 +0000 UTC m=+720.353085312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls") pod "error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" (UID: "c9980558-0b64-49ae-900a-c16cdc68794e") : secret "error-404-isvc-c21f8-predictor-serving-cert" not found
Jul 28 13:05:49.816244 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.816225    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9980558-0b64-49ae-900a-c16cdc68794e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:49.824326 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:49.824307    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pvh\" (UniqueName: \"kubernetes.io/projected/c9980558-0b64-49ae-900a-c16cdc68794e-kube-api-access-l2pvh\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:50.319662 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:50.319621    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:50.319835 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:05:50.319730    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-serving-cert: secret "error-404-isvc-c21f8-predictor-serving-cert" not found
Jul 28 13:05:50.319835 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:05:50.319811    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls podName:c9980558-0b64-49ae-900a-c16cdc68794e nodeName:}" failed. No retries permitted until 2026-07-28 13:05:51.319782055 +0000 UTC m=+721.357150940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls") pod "error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" (UID: "c9980558-0b64-49ae-900a-c16cdc68794e") : secret "error-404-isvc-c21f8-predictor-serving-cert" not found
Jul 28 13:05:50.619423 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:50.619343    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:05:50.656315 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:50.656290    2553 generic.go:356] "Generic (PLEG): container finished" podID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerID="8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf" exitCode=2
Jul 28 13:05:50.656442 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:50.656357    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" event={"ID":"2dd0601b-6fa6-407f-aaf2-b9741e3834c2","Type":"ContainerDied","Data":"8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf"}
Jul 28 13:05:51.327266 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:51.327230    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:51.329511 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:51.329492    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls\") pod \"error-404-isvc-c21f8-predictor-5c8d687554-v2gg7\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") " pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:51.500967 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:51.500933    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:51.626302 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:51.626279    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"]
Jul 28 13:05:51.628548 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:05:51.628510    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9980558_0b64_49ae_900a_c16cdc68794e.slice/crio-fbc528abae0bdb2f0ada8197a055b5acf3a6ca282d80487a79e3a0c748ac2ea0 WatchSource:0}: Error finding container fbc528abae0bdb2f0ada8197a055b5acf3a6ca282d80487a79e3a0c748ac2ea0: Status 404 returned error can't find the container with id fbc528abae0bdb2f0ada8197a055b5acf3a6ca282d80487a79e3a0c748ac2ea0
Jul 28 13:05:51.661961 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:51.661936    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" event={"ID":"c9980558-0b64-49ae-900a-c16cdc68794e","Type":"ContainerStarted","Data":"fbc528abae0bdb2f0ada8197a055b5acf3a6ca282d80487a79e3a0c748ac2ea0"}
Jul 28 13:05:52.378389 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.378365    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:05:52.435379 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.435345    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-proxy-tls\") pod \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") "
Jul 28 13:05:52.435561 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.435404    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-rbac-proxy-sar-config\") pod \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") "
Jul 28 13:05:52.435561 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.435453    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll9nb\" (UniqueName: \"kubernetes.io/projected/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-api-access-ll9nb\") pod \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\" (UID: \"2dd0601b-6fa6-407f-aaf2-b9741e3834c2\") "
Jul 28 13:05:52.435738 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.435712    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2dd0601b-6fa6-407f-aaf2-b9741e3834c2" (UID: "2dd0601b-6fa6-407f-aaf2-b9741e3834c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:05:52.437571 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.437548    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2dd0601b-6fa6-407f-aaf2-b9741e3834c2" (UID: "2dd0601b-6fa6-407f-aaf2-b9741e3834c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:05:52.437640 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.437592    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-api-access-ll9nb" (OuterVolumeSpecName: "kube-api-access-ll9nb") pod "2dd0601b-6fa6-407f-aaf2-b9741e3834c2" (UID: "2dd0601b-6fa6-407f-aaf2-b9741e3834c2"). InnerVolumeSpecName "kube-api-access-ll9nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:05:52.536577 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.536535    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll9nb\" (UniqueName: \"kubernetes.io/projected/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-api-access-ll9nb\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:05:52.536577 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.536566    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:05:52.536577 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.536579    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd0601b-6fa6-407f-aaf2-b9741e3834c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:05:52.668097 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.668065    2553 generic.go:356] "Generic (PLEG): container finished" podID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerID="c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8" exitCode=0
Jul 28 13:05:52.668553 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.668128    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"
Jul 28 13:05:52.668553 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.668156    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" event={"ID":"2dd0601b-6fa6-407f-aaf2-b9741e3834c2","Type":"ContainerDied","Data":"c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8"}
Jul 28 13:05:52.668553 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.668238    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" event={"ID":"2dd0601b-6fa6-407f-aaf2-b9741e3834c2","Type":"ContainerDied","Data":"ddc89a128a5a0dcb8028edd0655792d8a672f3ab2040b31ce43bbe4dba141d35"}
Jul 28 13:05:52.668553 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.668283    2553 scope.go:117] "RemoveContainer" containerID="8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf"
Jul 28 13:05:52.670498 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.670472    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" event={"ID":"c9980558-0b64-49ae-900a-c16cdc68794e","Type":"ContainerStarted","Data":"40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f"}
Jul 28 13:05:52.670605 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.670537    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" event={"ID":"c9980558-0b64-49ae-900a-c16cdc68794e","Type":"ContainerStarted","Data":"0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec"}
Jul 28 13:05:52.671277 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.671163    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:52.671277 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.671234    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:52.672468 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.672441    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 13:05:52.677463 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.677445    2553 scope.go:117] "RemoveContainer" containerID="c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8"
Jul 28 13:05:52.685198 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.685180    2553 scope.go:117] "RemoveContainer" containerID="8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf"
Jul 28 13:05:52.685457 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:05:52.685437    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf\": container with ID starting with 8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf not found: ID does not exist" containerID="8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf"
Jul 28 13:05:52.685514 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.685466    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf"} err="failed to get container status \"8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf\": rpc error: code = NotFound desc = could not find container \"8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf\": container with ID starting with 8bd114ab0de9fa79ba398c2a0b26b8a3b871c25efe0d54c5f10f26ea9f197ecf not found: ID does not exist"
Jul 28 13:05:52.685514 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.685484    2553 scope.go:117] "RemoveContainer" containerID="c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8"
Jul 28 13:05:52.685713 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:05:52.685696    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8\": container with ID starting with c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8 not found: ID does not exist" containerID="c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8"
Jul 28 13:05:52.685755 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.685719    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8"} err="failed to get container status \"c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8\": rpc error: code = NotFound desc = could not find container \"c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8\": container with ID starting with c84f14e0a5821ba5ed4d338013b9fbb5dbd968b520e63c5c83aa93f03c8397a8 not found: ID does not exist"
Jul 28 13:05:52.695757 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.695715    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" podStartSLOduration=3.695701552 podStartE2EDuration="3.695701552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:05:52.693608372 +0000 UTC m=+722.730977304" watchObservedRunningTime="2026-07-28 13:05:52.695701552 +0000 UTC m=+722.733070458"
Jul 28 13:05:52.711836 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.711806    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"]
Jul 28 13:05:52.717229 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:52.717189    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d"]
Jul 28 13:05:53.341512 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:53.341453    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 13:05:53.347488 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:53.347455    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aee94-predictor-6c8ccbb46b-q678d" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: i/o timeout"
Jul 28 13:05:53.471929 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:53.471897    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:05:53.675837 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:53.675785    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 13:05:54.476349 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:54.476307    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" path="/var/lib/kubelet/pods/2dd0601b-6fa6-407f-aaf2-b9741e3834c2/volumes"
Jul 28 13:05:58.680811 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:58.680771    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:05:58.681245 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:05:58.681218    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 13:06:08.681803 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:08.681754    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 13:06:15.242321 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:15.242278    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:06:15.533326 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:15.533248    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:06:18.681392 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:18.681354    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 13:06:25.373581 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.373530    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:06:25.526135 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.526098    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:06:25.538978 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.538951    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"]
Jul 28 13:06:25.539219 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.539196    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container" containerID="cri-o://3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6" gracePeriod=30
Jul 28 13:06:25.539302 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.539253    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kube-rbac-proxy" containerID="cri-o://b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97" gracePeriod=30
Jul 28 13:06:25.637037 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.636962    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"]
Jul 28 13:06:25.637286 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.637243    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container" containerID="cri-o://d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223" gracePeriod=30
Jul 28 13:06:25.637396 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.637280    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kube-rbac-proxy" containerID="cri-o://3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393" gracePeriod=30
Jul 28 13:06:25.653864 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.653835    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:06:25.753191 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.753161    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"]
Jul 28 13:06:25.753610 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.753595    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container"
Jul 28 13:06:25.753690 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.753612    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container"
Jul 28 13:06:25.753690 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.753621    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kube-rbac-proxy"
Jul 28 13:06:25.753690 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.753627    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kube-rbac-proxy"
Jul 28 13:06:25.753809 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.753696    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kube-rbac-proxy"
Jul 28 13:06:25.753809 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.753708    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd0601b-6fa6-407f-aaf2-b9741e3834c2" containerName="kserve-container"
Jul 28 13:06:25.756822 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.756787    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:25.758004 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.757980    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:06:25.759642 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.759623    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fec78-predictor-serving-cert\""
Jul 28 13:06:25.759742 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.759700    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fec78-kube-rbac-proxy-sar-config\""
Jul 28 13:06:25.774849 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.774824    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"]
Jul 28 13:06:25.824516 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.824488    2553 generic.go:356] "Generic (PLEG): container finished" podID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerID="3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393" exitCode=2
Jul 28 13:06:25.824641 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.824559    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" event={"ID":"5d250e58-bb57-4e03-bebe-f22ba62ea946","Type":"ContainerDied","Data":"3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393"}
Jul 28 13:06:25.826666 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.826646    2553 generic.go:356] "Generic (PLEG): container finished" podID="f773907b-7eee-4385-ae85-d2134155cd42" containerID="b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97" exitCode=2
Jul 28 13:06:25.826760 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.826698    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" event={"ID":"f773907b-7eee-4385-ae85-d2134155cd42","Type":"ContainerDied","Data":"b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97"}
Jul 28 13:06:25.847734 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.847710    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:06:25.908551 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.908492    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8de6e196-32d6-4107-902d-1062cf141226-proxy-tls\") pod \"error-404-isvc-fec78-predictor-85b74f59f6-hkvv9\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") " pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:25.908551 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.908540    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8d9\" (UniqueName: \"kubernetes.io/projected/8de6e196-32d6-4107-902d-1062cf141226-kube-api-access-5r8d9\") pod \"error-404-isvc-fec78-predictor-85b74f59f6-hkvv9\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") " pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:25.908675 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:25.908628    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8de6e196-32d6-4107-902d-1062cf141226-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fec78-predictor-85b74f59f6-hkvv9\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") " pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.009730 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.009696    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8de6e196-32d6-4107-902d-1062cf141226-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fec78-predictor-85b74f59f6-hkvv9\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") " pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.009883 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.009744    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8de6e196-32d6-4107-902d-1062cf141226-proxy-tls\") pod \"error-404-isvc-fec78-predictor-85b74f59f6-hkvv9\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") " pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.009883 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.009785    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8d9\" (UniqueName: \"kubernetes.io/projected/8de6e196-32d6-4107-902d-1062cf141226-kube-api-access-5r8d9\") pod \"error-404-isvc-fec78-predictor-85b74f59f6-hkvv9\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") " pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.010464 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.010439    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8de6e196-32d6-4107-902d-1062cf141226-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fec78-predictor-85b74f59f6-hkvv9\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") " pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.012165 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.012145    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8de6e196-32d6-4107-902d-1062cf141226-proxy-tls\") pod \"error-404-isvc-fec78-predictor-85b74f59f6-hkvv9\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") " pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.021289 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.021266    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8d9\" (UniqueName: \"kubernetes.io/projected/8de6e196-32d6-4107-902d-1062cf141226-kube-api-access-5r8d9\") pod \"error-404-isvc-fec78-predictor-85b74f59f6-hkvv9\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") " pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.066758 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.066735    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.200151 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.200069    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"]
Jul 28 13:06:26.202624 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:06:26.202599    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de6e196_32d6_4107_902d_1062cf141226.slice/crio-8c73f58cf87bfbef6f7e42999613c5d5e4c731c97e712d8d5beb0f1cbec52541 WatchSource:0}: Error finding container 8c73f58cf87bfbef6f7e42999613c5d5e4c731c97e712d8d5beb0f1cbec52541: Status 404 returned error can't find the container with id 8c73f58cf87bfbef6f7e42999613c5d5e4c731c97e712d8d5beb0f1cbec52541
Jul 28 13:06:26.832503 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.832466    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" event={"ID":"8de6e196-32d6-4107-902d-1062cf141226","Type":"ContainerStarted","Data":"9abb5d4ad8b156d922298a337b2a20f34436f050a10b94c77414e9f63ebe30ea"}
Jul 28 13:06:26.833031 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.832530    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" event={"ID":"8de6e196-32d6-4107-902d-1062cf141226","Type":"ContainerStarted","Data":"8834c7da6e90842bb2d848c46b852569bf623812045215ca7676b00cf338b91f"}
Jul 28 13:06:26.833031 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.832561    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" event={"ID":"8de6e196-32d6-4107-902d-1062cf141226","Type":"ContainerStarted","Data":"8c73f58cf87bfbef6f7e42999613c5d5e4c731c97e712d8d5beb0f1cbec52541"}
Jul 28 13:06:26.833031 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.832602    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.833031 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.832636    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:26.834039 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.834014    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 13:06:26.853743 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:26.853689    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podStartSLOduration=1.8536735549999999 podStartE2EDuration="1.853673555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:06:26.852833413 +0000 UTC m=+756.890202322" watchObservedRunningTime="2026-07-28 13:06:26.853673555 +0000 UTC m=+756.891042463"
Jul 28 13:06:27.341586 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:27.341548    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 13:06:27.843204 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:27.843163    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 13:06:28.420653 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:28.420608    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 13:06:28.425991 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:28.425959    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 13:06:28.681730 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:28.681631    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 13:06:29.583808 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.583764    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:06:29.740649 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.740623    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d250e58-bb57-4e03-bebe-f22ba62ea946-kserve-provision-location\") pod \"5d250e58-bb57-4e03-bebe-f22ba62ea946\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") "
Jul 28 13:06:29.740855 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.740664    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhgm5\" (UniqueName: \"kubernetes.io/projected/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-api-access-nhgm5\") pod \"5d250e58-bb57-4e03-bebe-f22ba62ea946\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") "
Jul 28 13:06:29.740855 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.740711    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-cabundle-cert\") pod \"5d250e58-bb57-4e03-bebe-f22ba62ea946\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") "
Jul 28 13:06:29.740855 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.740729    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d250e58-bb57-4e03-bebe-f22ba62ea946-proxy-tls\") pod \"5d250e58-bb57-4e03-bebe-f22ba62ea946\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") "
Jul 28 13:06:29.740855 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.740747    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-rbac-proxy-sar-config\") pod \"5d250e58-bb57-4e03-bebe-f22ba62ea946\" (UID: \"5d250e58-bb57-4e03-bebe-f22ba62ea946\") "
Jul 28 13:06:29.741142 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.741005    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d250e58-bb57-4e03-bebe-f22ba62ea946-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d250e58-bb57-4e03-bebe-f22ba62ea946" (UID: "5d250e58-bb57-4e03-bebe-f22ba62ea946"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:06:29.741142 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.741083    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d250e58-bb57-4e03-bebe-f22ba62ea946" (UID: "5d250e58-bb57-4e03-bebe-f22ba62ea946"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:06:29.741224 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.741169    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d250e58-bb57-4e03-bebe-f22ba62ea946" (UID: "5d250e58-bb57-4e03-bebe-f22ba62ea946"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:06:29.742813 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.742778    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d250e58-bb57-4e03-bebe-f22ba62ea946-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d250e58-bb57-4e03-bebe-f22ba62ea946" (UID: "5d250e58-bb57-4e03-bebe-f22ba62ea946"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:06:29.742959 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.742940    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-api-access-nhgm5" (OuterVolumeSpecName: "kube-api-access-nhgm5") pod "5d250e58-bb57-4e03-bebe-f22ba62ea946" (UID: "5d250e58-bb57-4e03-bebe-f22ba62ea946"). InnerVolumeSpecName "kube-api-access-nhgm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:06:29.841511 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.841480    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-cabundle-cert\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:29.841511 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.841507    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d250e58-bb57-4e03-bebe-f22ba62ea946-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:29.841661 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.841517    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:29.841661 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.841527    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d250e58-bb57-4e03-bebe-f22ba62ea946-kserve-provision-location\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:29.841661 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.841536    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhgm5\" (UniqueName: \"kubernetes.io/projected/5d250e58-bb57-4e03-bebe-f22ba62ea946-kube-api-access-nhgm5\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:29.853159 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.853128    2553 generic.go:356] "Generic (PLEG): container finished" podID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerID="d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223" exitCode=0
Jul 28 13:06:29.853267 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.853168    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" event={"ID":"5d250e58-bb57-4e03-bebe-f22ba62ea946","Type":"ContainerDied","Data":"d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223"}
Jul 28 13:06:29.853267 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.853181    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"
Jul 28 13:06:29.853267 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.853233    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27" event={"ID":"5d250e58-bb57-4e03-bebe-f22ba62ea946","Type":"ContainerDied","Data":"d47d833aee362227c2e1ac04270198f36500721e0db3443e5af7b03679cf821e"}
Jul 28 13:06:29.853267 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.853262    2553 scope.go:117] "RemoveContainer" containerID="3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393"
Jul 28 13:06:29.861709 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.861688    2553 scope.go:117] "RemoveContainer" containerID="d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223"
Jul 28 13:06:29.869103 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.869084    2553 scope.go:117] "RemoveContainer" containerID="638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb"
Jul 28 13:06:29.874938 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.874912    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"]
Jul 28 13:06:29.876659 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.876643    2553 scope.go:117] "RemoveContainer" containerID="3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393"
Jul 28 13:06:29.876992 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:06:29.876959    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393\": container with ID starting with 3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393 not found: ID does not exist" containerID="3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393"
Jul 28 13:06:29.877129 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.877002    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393"} err="failed to get container status \"3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393\": rpc error: code = NotFound desc = could not find container \"3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393\": container with ID starting with 3694e6f4209e96b4e63e625f4e5d3c51a9b9b3e8c916792da775c5f768d79393 not found: ID does not exist"
Jul 28 13:06:29.877129 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.877025    2553 scope.go:117] "RemoveContainer" containerID="d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223"
Jul 28 13:06:29.877433 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:06:29.877406    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223\": container with ID starting with d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223 not found: ID does not exist" containerID="d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223"
Jul 28 13:06:29.877514 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.877441    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223"} err="failed to get container status \"d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223\": rpc error: code = NotFound desc = could not find container \"d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223\": container with ID starting with d64ee49cb207bcbededd072edd1c6a8651fb308fda033967dfda5420404c2223 not found: ID does not exist"
Jul 28 13:06:29.877514 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.877463    2553 scope.go:117] "RemoveContainer" containerID="638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb"
Jul 28 13:06:29.877759 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:06:29.877738    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb\": container with ID starting with 638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb not found: ID does not exist" containerID="638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb"
Jul 28 13:06:29.877834 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.877767    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb"} err="failed to get container status \"638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb\": rpc error: code = NotFound desc = could not find container \"638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb\": container with ID starting with 638fea192d2cebf50fcbff8cd74b7ce6dfbd696cee2a9eb8b32e400aea8d8ddb not found: ID does not exist"
Jul 28 13:06:29.879178 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:29.879162    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-98h27"]
Jul 28 13:06:30.480073 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.480048    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" path="/var/lib/kubelet/pods/5d250e58-bb57-4e03-bebe-f22ba62ea946/volumes"
Jul 28 13:06:30.492049 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.492031    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:06:30.647974 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.647900    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-kube-rbac-proxy-sar-config\") pod \"f773907b-7eee-4385-ae85-d2134155cd42\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") "
Jul 28 13:06:30.647974 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.647937    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9npbh\" (UniqueName: \"kubernetes.io/projected/f773907b-7eee-4385-ae85-d2134155cd42-kube-api-access-9npbh\") pod \"f773907b-7eee-4385-ae85-d2134155cd42\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") "
Jul 28 13:06:30.648451 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.647987    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f773907b-7eee-4385-ae85-d2134155cd42-kserve-provision-location\") pod \"f773907b-7eee-4385-ae85-d2134155cd42\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") "
Jul 28 13:06:30.648451 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.648036    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f773907b-7eee-4385-ae85-d2134155cd42-proxy-tls\") pod \"f773907b-7eee-4385-ae85-d2134155cd42\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") "
Jul 28 13:06:30.648451 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.648068    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-cabundle-cert\") pod \"f773907b-7eee-4385-ae85-d2134155cd42\" (UID: \"f773907b-7eee-4385-ae85-d2134155cd42\") "
Jul 28 13:06:30.648451 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.648331    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f773907b-7eee-4385-ae85-d2134155cd42" (UID: "f773907b-7eee-4385-ae85-d2134155cd42"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:06:30.648451 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.648368    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f773907b-7eee-4385-ae85-d2134155cd42-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f773907b-7eee-4385-ae85-d2134155cd42" (UID: "f773907b-7eee-4385-ae85-d2134155cd42"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:06:30.648451 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.648429    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f773907b-7eee-4385-ae85-d2134155cd42" (UID: "f773907b-7eee-4385-ae85-d2134155cd42"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:06:30.650074 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.650046    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f773907b-7eee-4385-ae85-d2134155cd42-kube-api-access-9npbh" (OuterVolumeSpecName: "kube-api-access-9npbh") pod "f773907b-7eee-4385-ae85-d2134155cd42" (UID: "f773907b-7eee-4385-ae85-d2134155cd42"). InnerVolumeSpecName "kube-api-access-9npbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:06:30.650183 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.650081    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f773907b-7eee-4385-ae85-d2134155cd42-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f773907b-7eee-4385-ae85-d2134155cd42" (UID: "f773907b-7eee-4385-ae85-d2134155cd42"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:06:30.749656 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.749628    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f773907b-7eee-4385-ae85-d2134155cd42-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:30.749656 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.749651    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-cabundle-cert\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:30.749656 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.749662    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f773907b-7eee-4385-ae85-d2134155cd42-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:30.749885 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.749670    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9npbh\" (UniqueName: \"kubernetes.io/projected/f773907b-7eee-4385-ae85-d2134155cd42-kube-api-access-9npbh\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:30.749885 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.749690    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f773907b-7eee-4385-ae85-d2134155cd42-kserve-provision-location\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:06:30.860050 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.860021    2553 generic.go:356] "Generic (PLEG): container finished" podID="f773907b-7eee-4385-ae85-d2134155cd42" containerID="3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6" exitCode=0
Jul 28 13:06:30.860193 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.860066    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" event={"ID":"f773907b-7eee-4385-ae85-d2134155cd42","Type":"ContainerDied","Data":"3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6"}
Jul 28 13:06:30.860193 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.860079    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"
Jul 28 13:06:30.860193 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.860117    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd" event={"ID":"f773907b-7eee-4385-ae85-d2134155cd42","Type":"ContainerDied","Data":"157db00c98292f5b51e4dd3dba323795313c59b7d1790579f18b8ea6bc8af27e"}
Jul 28 13:06:30.860193 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.860144    2553 scope.go:117] "RemoveContainer" containerID="b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97"
Jul 28 13:06:30.868454 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.868439    2553 scope.go:117] "RemoveContainer" containerID="3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6"
Jul 28 13:06:30.875830 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.875810    2553 scope.go:117] "RemoveContainer" containerID="6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66"
Jul 28 13:06:30.883158 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.883103    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"]
Jul 28 13:06:30.883232 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.883177    2553 scope.go:117] "RemoveContainer" containerID="b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97"
Jul 28 13:06:30.883445 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:06:30.883429    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97\": container with ID starting with b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97 not found: ID does not exist" containerID="b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97"
Jul 28 13:06:30.883564 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.883455    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97"} err="failed to get container status \"b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97\": rpc error: code = NotFound desc = could not find container \"b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97\": container with ID starting with b1e58fd0aef4b38afbc3df73606fc209fac1096530d6039bc85c36eed4187f97 not found: ID does not exist"
Jul 28 13:06:30.883564 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.883475    2553 scope.go:117] "RemoveContainer" containerID="3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6"
Jul 28 13:06:30.883725 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:06:30.883705    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6\": container with ID starting with 3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6 not found: ID does not exist" containerID="3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6"
Jul 28 13:06:30.883768 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.883730    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6"} err="failed to get container status \"3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6\": rpc error: code = NotFound desc = could not find container \"3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6\": container with ID starting with 3e8ec57f62f40651c15d4682aee23dd4586993c4efee1467652113e2b38668f6 not found: ID does not exist"
Jul 28 13:06:30.883768 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.883745    2553 scope.go:117] "RemoveContainer" containerID="6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66"
Jul 28 13:06:30.883958 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:06:30.883942    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66\": container with ID starting with 6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66 not found: ID does not exist" containerID="6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66"
Jul 28 13:06:30.883996 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.883962    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66"} err="failed to get container status \"6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66\": rpc error: code = NotFound desc = could not find container \"6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66\": container with ID starting with 6f286038e32fcd0af05987dad7cf9fa8352d64f69d828eb900e102c09b9c1f66 not found: ID does not exist"
Jul 28 13:06:30.889141 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:30.889121    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-s8ssd"]
Jul 28 13:06:32.475898 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:32.475860    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f773907b-7eee-4385-ae85-d2134155cd42" path="/var/lib/kubelet/pods/f773907b-7eee-4385-ae85-d2134155cd42/volumes"
Jul 28 13:06:32.848186 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:32.848104    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:06:32.848704 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:32.848669    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 13:06:38.681931 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:38.681905    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:06:42.849272 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:42.849227    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 13:06:49.643817 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:49.643759    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:06:49.945351 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:49.945315    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:06:52.848676 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:06:52.848638    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 13:07:02.849533 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:07:02.849491    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 13:07:12.850172 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:07:12.850103    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:07:25.585943 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:07:25.585908    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:07:26.216474 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:07:26.216435    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:07:55.142954 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:07:55.142917    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:07:55.244073 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:07:55.244043    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:08:28.989121 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:08:28.989076    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:08:29.089024 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:08:29.088997    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:11:32.813836 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:11:32.813785    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:12:19.769256 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:12:19.769202    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:12:19.821480 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:12:19.821425    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:13:39.043536 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:13:39.043421    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:04.362903 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.362824    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:04.499666 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.499634    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:04.601432 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.601403    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:04.676691 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.676666    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:04.681350 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681330    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"]
Jul 28 13:15:04.681782 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681765    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681787    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681808    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="storage-initializer"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681816    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="storage-initializer"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681826    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kube-rbac-proxy"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681831    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kube-rbac-proxy"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681842    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="storage-initializer"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681850    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="storage-initializer"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681860    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kube-rbac-proxy"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681865    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kube-rbac-proxy"
Jul 28 13:15:04.681870 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681873    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container"
Jul 28 13:15:04.682154 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681878    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container"
Jul 28 13:15:04.682154 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681941    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kube-rbac-proxy"
Jul 28 13:15:04.682154 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681948    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d250e58-bb57-4e03-bebe-f22ba62ea946" containerName="kserve-container"
Jul 28 13:15:04.682154 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681955    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kube-rbac-proxy"
Jul 28 13:15:04.682154 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.681961    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f773907b-7eee-4385-ae85-d2134155cd42" containerName="kserve-container"
Jul 28 13:15:04.685094 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.685077    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:04.688086 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.688069    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dec07-predictor-serving-cert\""
Jul 28 13:15:04.688086 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.688080    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dec07-kube-rbac-proxy-sar-config\""
Jul 28 13:15:04.695542 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.695519    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"]
Jul 28 13:15:04.721366 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.721342    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"]
Jul 28 13:15:04.721552 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.721526    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container" containerID="cri-o://0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec" gracePeriod=30
Jul 28 13:15:04.721693 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.721559    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kube-rbac-proxy" containerID="cri-o://40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f" gracePeriod=30
Jul 28 13:15:04.745821 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.745779    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:04.784787 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.784767    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d878539b-0f35-4694-879b-349f708a32b5-proxy-tls\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:04.784896 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.784813    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d878539b-0f35-4694-879b-349f708a32b5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:04.784942 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.784901    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8sg\" (UniqueName: \"kubernetes.io/projected/d878539b-0f35-4694-879b-349f708a32b5-kube-api-access-dl8sg\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:04.885409 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.885384    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d878539b-0f35-4694-879b-349f708a32b5-proxy-tls\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:04.885512 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.885424    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d878539b-0f35-4694-879b-349f708a32b5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:04.885512 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.885468    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8sg\" (UniqueName: \"kubernetes.io/projected/d878539b-0f35-4694-879b-349f708a32b5-kube-api-access-dl8sg\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:04.885622 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:15:04.885549    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-dec07-predictor-serving-cert: secret "error-404-isvc-dec07-predictor-serving-cert" not found
Jul 28 13:15:04.885622 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:15:04.885621    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d878539b-0f35-4694-879b-349f708a32b5-proxy-tls podName:d878539b-0f35-4694-879b-349f708a32b5 nodeName:}" failed. No retries permitted until 2026-07-28 13:15:05.385601113 +0000 UTC m=+1275.422970013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d878539b-0f35-4694-879b-349f708a32b5-proxy-tls") pod "error-404-isvc-dec07-predictor-799fbb6756-wjkbv" (UID: "d878539b-0f35-4694-879b-349f708a32b5") : secret "error-404-isvc-dec07-predictor-serving-cert" not found
Jul 28 13:15:04.886068 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.886048    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d878539b-0f35-4694-879b-349f708a32b5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:04.894543 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.894524    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8sg\" (UniqueName: \"kubernetes.io/projected/d878539b-0f35-4694-879b-349f708a32b5-kube-api-access-dl8sg\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:04.971424 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.971371    2553 generic.go:356] "Generic (PLEG): container finished" podID="c9980558-0b64-49ae-900a-c16cdc68794e" containerID="40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f" exitCode=2
Jul 28 13:15:04.971510 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:04.971441    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" event={"ID":"c9980558-0b64-49ae-900a-c16cdc68794e","Type":"ContainerDied","Data":"40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f"}
Jul 28 13:15:05.072677 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.072655    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:05.390220 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.390138    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d878539b-0f35-4694-879b-349f708a32b5-proxy-tls\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:05.392409 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.392386    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d878539b-0f35-4694-879b-349f708a32b5-proxy-tls\") pod \"error-404-isvc-dec07-predictor-799fbb6756-wjkbv\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") " pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:05.468410 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.468383    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:05.595194 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.595172    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:05.724085 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.724060    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"]
Jul 28 13:15:05.725820 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:15:05.725782    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd878539b_0f35_4694_879b_349f708a32b5.slice/crio-c6ec6618b4bbd196d63b346defff9cd2472002794a085ee4f3d618efa5459b82 WatchSource:0}: Error finding container c6ec6618b4bbd196d63b346defff9cd2472002794a085ee4f3d618efa5459b82: Status 404 returned error can't find the container with id c6ec6618b4bbd196d63b346defff9cd2472002794a085ee4f3d618efa5459b82
Jul 28 13:15:05.977758 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.977722    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" event={"ID":"d878539b-0f35-4694-879b-349f708a32b5","Type":"ContainerStarted","Data":"eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132"}
Jul 28 13:15:05.977971 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.977782    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" event={"ID":"d878539b-0f35-4694-879b-349f708a32b5","Type":"ContainerStarted","Data":"c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec"}
Jul 28 13:15:05.977971 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.977925    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:05.977971 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.977950    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:05.977971 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.977967    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" event={"ID":"d878539b-0f35-4694-879b-349f708a32b5","Type":"ContainerStarted","Data":"c6ec6618b4bbd196d63b346defff9cd2472002794a085ee4f3d618efa5459b82"}
Jul 28 13:15:05.979342 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.979315    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 13:15:05.996595 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:05.996538    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podStartSLOduration=1.996524327 podStartE2EDuration="1.996524327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:15:05.995512311 +0000 UTC m=+1276.032881220" watchObservedRunningTime="2026-07-28 13:15:05.996524327 +0000 UTC m=+1276.033893234"
Jul 28 13:15:06.981809 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:06.981757    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 13:15:07.567220 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.567198    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:15:07.710559 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.710535    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls\") pod \"c9980558-0b64-49ae-900a-c16cdc68794e\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") "
Jul 28 13:15:07.710731 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.710578    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9980558-0b64-49ae-900a-c16cdc68794e-kube-rbac-proxy-sar-config\") pod \"c9980558-0b64-49ae-900a-c16cdc68794e\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") "
Jul 28 13:15:07.710731 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.710657    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2pvh\" (UniqueName: \"kubernetes.io/projected/c9980558-0b64-49ae-900a-c16cdc68794e-kube-api-access-l2pvh\") pod \"c9980558-0b64-49ae-900a-c16cdc68794e\" (UID: \"c9980558-0b64-49ae-900a-c16cdc68794e\") "
Jul 28 13:15:07.710912 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.710887    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9980558-0b64-49ae-900a-c16cdc68794e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c9980558-0b64-49ae-900a-c16cdc68794e" (UID: "c9980558-0b64-49ae-900a-c16cdc68794e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:15:07.712321 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.712301    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c9980558-0b64-49ae-900a-c16cdc68794e" (UID: "c9980558-0b64-49ae-900a-c16cdc68794e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:15:07.712436 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.712415    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9980558-0b64-49ae-900a-c16cdc68794e-kube-api-access-l2pvh" (OuterVolumeSpecName: "kube-api-access-l2pvh") pod "c9980558-0b64-49ae-900a-c16cdc68794e" (UID: "c9980558-0b64-49ae-900a-c16cdc68794e"). InnerVolumeSpecName "kube-api-access-l2pvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:15:07.812144 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.812121    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2pvh\" (UniqueName: \"kubernetes.io/projected/c9980558-0b64-49ae-900a-c16cdc68794e-kube-api-access-l2pvh\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:15:07.812144 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.812141    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9980558-0b64-49ae-900a-c16cdc68794e-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:15:07.812269 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.812150    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9980558-0b64-49ae-900a-c16cdc68794e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:15:07.986819 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.986726    2553 generic.go:356] "Generic (PLEG): container finished" podID="c9980558-0b64-49ae-900a-c16cdc68794e" containerID="0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec" exitCode=0
Jul 28 13:15:07.987305 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.986813    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"
Jul 28 13:15:07.987305 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.986816    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" event={"ID":"c9980558-0b64-49ae-900a-c16cdc68794e","Type":"ContainerDied","Data":"0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec"}
Jul 28 13:15:07.987305 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.986879    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7" event={"ID":"c9980558-0b64-49ae-900a-c16cdc68794e","Type":"ContainerDied","Data":"fbc528abae0bdb2f0ada8197a055b5acf3a6ca282d80487a79e3a0c748ac2ea0"}
Jul 28 13:15:07.987305 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.986904    2553 scope.go:117] "RemoveContainer" containerID="40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f"
Jul 28 13:15:07.995472 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:07.995459    2553 scope.go:117] "RemoveContainer" containerID="0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec"
Jul 28 13:15:08.004016 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:08.004000    2553 scope.go:117] "RemoveContainer" containerID="40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f"
Jul 28 13:15:08.004257 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:15:08.004240    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f\": container with ID starting with 40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f not found: ID does not exist" containerID="40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f"
Jul 28 13:15:08.004319 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:08.004263    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f"} err="failed to get container status \"40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f\": rpc error: code = NotFound desc = could not find container \"40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f\": container with ID starting with 40f4dc5c920749b1c0c2aa31ece91710ecdba3198a15c68baeb2414cc63e812f not found: ID does not exist"
Jul 28 13:15:08.004319 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:08.004276    2553 scope.go:117] "RemoveContainer" containerID="0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec"
Jul 28 13:15:08.004499 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:15:08.004483    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec\": container with ID starting with 0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec not found: ID does not exist" containerID="0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec"
Jul 28 13:15:08.004537 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:08.004503    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec"} err="failed to get container status \"0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec\": rpc error: code = NotFound desc = could not find container \"0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec\": container with ID starting with 0d6583269193f6e39aeb31aa4fa63ebc69f72e89fcb62fdce1efda36bf6630ec not found: ID does not exist"
Jul 28 13:15:08.010078 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:08.010057    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"]
Jul 28 13:15:08.015957 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:08.015931    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c21f8-predictor-5c8d687554-v2gg7"]
Jul 28 13:15:08.475944 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:08.475913    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" path="/var/lib/kubelet/pods/c9980558-0b64-49ae-900a-c16cdc68794e/volumes"
Jul 28 13:15:11.985668 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:11.985637    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:11.986133 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:11.986104    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 13:15:21.986179 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:21.986140    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 13:15:31.986584 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:31.986546    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 13:15:40.211402 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.211371    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:40.285477 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.285447    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"]
Jul 28 13:15:40.285681 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.285654    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container" containerID="cri-o://8834c7da6e90842bb2d848c46b852569bf623812045215ca7676b00cf338b91f" gracePeriod=30
Jul 28 13:15:40.285774 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.285699    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kube-rbac-proxy" containerID="cri-o://9abb5d4ad8b156d922298a337b2a20f34436f050a10b94c77414e9f63ebe30ea" gracePeriod=30
Jul 28 13:15:40.300019 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.299980    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:40.397511 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.397484    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:40.463363 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.463281    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:40.518891 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.518864    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"]
Jul 28 13:15:40.519206 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.519194    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kube-rbac-proxy"
Jul 28 13:15:40.519206 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.519207    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kube-rbac-proxy"
Jul 28 13:15:40.519290 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.519225    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container"
Jul 28 13:15:40.519290 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.519231    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container"
Jul 28 13:15:40.519357 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.519306    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kube-rbac-proxy"
Jul 28 13:15:40.519357 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.519316    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9980558-0b64-49ae-900a-c16cdc68794e" containerName="kserve-container"
Jul 28 13:15:40.522082 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.522067    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.524931 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.524910    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7ff2e-predictor-serving-cert\""
Jul 28 13:15:40.525038 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.524911    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7ff2e-kube-rbac-proxy-sar-config\""
Jul 28 13:15:40.531115 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.531096    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"]
Jul 28 13:15:40.556923 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.556902    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:40.569583 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.569558    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-proxy-tls\") pod \"error-404-isvc-7ff2e-predictor-7484d48446-grrpl\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") " pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.569698 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.569613    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztg8w\" (UniqueName: \"kubernetes.io/projected/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-api-access-ztg8w\") pod \"error-404-isvc-7ff2e-predictor-7484d48446-grrpl\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") " pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.569698 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.569667    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ff2e-predictor-7484d48446-grrpl\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") " pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.622132 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.622100    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:15:40.670383 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.670362    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztg8w\" (UniqueName: \"kubernetes.io/projected/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-api-access-ztg8w\") pod \"error-404-isvc-7ff2e-predictor-7484d48446-grrpl\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") " pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.670494 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.670390    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ff2e-predictor-7484d48446-grrpl\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") " pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.670596 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.670579    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-proxy-tls\") pod \"error-404-isvc-7ff2e-predictor-7484d48446-grrpl\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") " pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.671134 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.671118    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ff2e-predictor-7484d48446-grrpl\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") " pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.672865 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.672845    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-proxy-tls\") pod \"error-404-isvc-7ff2e-predictor-7484d48446-grrpl\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") " pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.679368 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.679342    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztg8w\" (UniqueName: \"kubernetes.io/projected/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-api-access-ztg8w\") pod \"error-404-isvc-7ff2e-predictor-7484d48446-grrpl\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") " pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.831832 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.831750    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:40.962602 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:40.962575    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"]
Jul 28 13:15:40.964326 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:15:40.964302    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d4ffd3_1013_4d9f_ac54_bc8b5fdf8f28.slice/crio-e758c8c99b7ebb5e4378f8e0205c8127750ebdb0f3039a0165de3e2fa7c4a5fe WatchSource:0}: Error finding container e758c8c99b7ebb5e4378f8e0205c8127750ebdb0f3039a0165de3e2fa7c4a5fe: Status 404 returned error can't find the container with id e758c8c99b7ebb5e4378f8e0205c8127750ebdb0f3039a0165de3e2fa7c4a5fe
Jul 28 13:15:41.121770 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.121739    2553 generic.go:356] "Generic (PLEG): container finished" podID="8de6e196-32d6-4107-902d-1062cf141226" containerID="9abb5d4ad8b156d922298a337b2a20f34436f050a10b94c77414e9f63ebe30ea" exitCode=2
Jul 28 13:15:41.121892 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.121829    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" event={"ID":"8de6e196-32d6-4107-902d-1062cf141226","Type":"ContainerDied","Data":"9abb5d4ad8b156d922298a337b2a20f34436f050a10b94c77414e9f63ebe30ea"}
Jul 28 13:15:41.123543 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.123521    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" event={"ID":"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28","Type":"ContainerStarted","Data":"479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5"}
Jul 28 13:15:41.123648 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.123562    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" event={"ID":"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28","Type":"ContainerStarted","Data":"5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409"}
Jul 28 13:15:41.123648 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.123581    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" event={"ID":"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28","Type":"ContainerStarted","Data":"e758c8c99b7ebb5e4378f8e0205c8127750ebdb0f3039a0165de3e2fa7c4a5fe"}
Jul 28 13:15:41.123738 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.123701    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:41.123778 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.123746    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:41.125066 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.125031    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 13:15:41.140732 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.140688    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podStartSLOduration=1.140672814 podStartE2EDuration="1.140672814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:15:41.140486606 +0000 UTC m=+1311.177855543" watchObservedRunningTime="2026-07-28 13:15:41.140672814 +0000 UTC m=+1311.178041734"
Jul 28 13:15:41.986671 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:41.986633    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 13:15:42.127788 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:42.127749    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 13:15:42.844158 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:42.844110    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 28 13:15:42.849535 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:42.849503    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 13:15:43.133106 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.133078    2553 generic.go:356] "Generic (PLEG): container finished" podID="8de6e196-32d6-4107-902d-1062cf141226" containerID="8834c7da6e90842bb2d848c46b852569bf623812045215ca7676b00cf338b91f" exitCode=0
Jul 28 13:15:43.133434 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.133141    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" event={"ID":"8de6e196-32d6-4107-902d-1062cf141226","Type":"ContainerDied","Data":"8834c7da6e90842bb2d848c46b852569bf623812045215ca7676b00cf338b91f"}
Jul 28 13:15:43.530745 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.530721    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:15:43.596828 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.596786    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8de6e196-32d6-4107-902d-1062cf141226-proxy-tls\") pod \"8de6e196-32d6-4107-902d-1062cf141226\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") "
Jul 28 13:15:43.596971 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.596844    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r8d9\" (UniqueName: \"kubernetes.io/projected/8de6e196-32d6-4107-902d-1062cf141226-kube-api-access-5r8d9\") pod \"8de6e196-32d6-4107-902d-1062cf141226\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") "
Jul 28 13:15:43.596971 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.596870    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8de6e196-32d6-4107-902d-1062cf141226-kube-rbac-proxy-sar-config\") pod \"8de6e196-32d6-4107-902d-1062cf141226\" (UID: \"8de6e196-32d6-4107-902d-1062cf141226\") "
Jul 28 13:15:43.597270 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.597246    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de6e196-32d6-4107-902d-1062cf141226-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8de6e196-32d6-4107-902d-1062cf141226" (UID: "8de6e196-32d6-4107-902d-1062cf141226"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:15:43.598615 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.598593    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de6e196-32d6-4107-902d-1062cf141226-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8de6e196-32d6-4107-902d-1062cf141226" (UID: "8de6e196-32d6-4107-902d-1062cf141226"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:15:43.598928 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.598912    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de6e196-32d6-4107-902d-1062cf141226-kube-api-access-5r8d9" (OuterVolumeSpecName: "kube-api-access-5r8d9") pod "8de6e196-32d6-4107-902d-1062cf141226" (UID: "8de6e196-32d6-4107-902d-1062cf141226"). InnerVolumeSpecName "kube-api-access-5r8d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:15:43.697548 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.697527    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8de6e196-32d6-4107-902d-1062cf141226-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:15:43.697548 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.697547    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r8d9\" (UniqueName: \"kubernetes.io/projected/8de6e196-32d6-4107-902d-1062cf141226-kube-api-access-5r8d9\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:15:43.697671 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:43.697557    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8de6e196-32d6-4107-902d-1062cf141226-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:15:44.139220 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:44.139152    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"
Jul 28 13:15:44.139569 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:44.139144    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9" event={"ID":"8de6e196-32d6-4107-902d-1062cf141226","Type":"ContainerDied","Data":"8c73f58cf87bfbef6f7e42999613c5d5e4c731c97e712d8d5beb0f1cbec52541"}
Jul 28 13:15:44.139569 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:44.139291    2553 scope.go:117] "RemoveContainer" containerID="9abb5d4ad8b156d922298a337b2a20f34436f050a10b94c77414e9f63ebe30ea"
Jul 28 13:15:44.148450 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:44.148437    2553 scope.go:117] "RemoveContainer" containerID="8834c7da6e90842bb2d848c46b852569bf623812045215ca7676b00cf338b91f"
Jul 28 13:15:44.162128 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:44.162106    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"]
Jul 28 13:15:44.165546 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:44.165525    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fec78-predictor-85b74f59f6-hkvv9"]
Jul 28 13:15:44.475577 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:44.475543    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de6e196-32d6-4107-902d-1062cf141226" path="/var/lib/kubelet/pods/8de6e196-32d6-4107-902d-1062cf141226/volumes"
Jul 28 13:15:47.132459 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:47.132434    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:15:47.132867 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:47.132836    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 13:15:51.987465 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:51.987438    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:15:57.133142 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:15:57.133102    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 13:16:04.610073 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:04.610044    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:04.712296 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:04.712265    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:07.133415 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:07.133376    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 13:16:14.695620 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.695589    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:14.822030 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.822000    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:14.847021 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.846987    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"]
Jul 28 13:16:14.847267 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.847242    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container" containerID="cri-o://c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec" gracePeriod=30
Jul 28 13:16:14.847365 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.847278    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kube-rbac-proxy" containerID="cri-o://eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132" gracePeriod=30
Jul 28 13:16:14.895841 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.895820    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:14.959162 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.959100    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"]
Jul 28 13:16:14.959499 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.959484    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kube-rbac-proxy"
Jul 28 13:16:14.959542 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.959501    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kube-rbac-proxy"
Jul 28 13:16:14.959542 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.959518    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container"
Jul 28 13:16:14.959542 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.959524    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container"
Jul 28 13:16:14.959784 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.959577    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kserve-container"
Jul 28 13:16:14.959784 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.959584    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de6e196-32d6-4107-902d-1062cf141226" containerName="kube-rbac-proxy"
Jul 28 13:16:14.962579 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.962557    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:14.963235 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.963212    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:14.965434 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.965415    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7cae9-predictor-serving-cert\""
Jul 28 13:16:14.965552 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.965435    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7cae9-kube-rbac-proxy-sar-config\""
Jul 28 13:16:14.972261 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:14.972240    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"]
Jul 28 13:16:15.058955 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.058928    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sztk\" (UniqueName: \"kubernetes.io/projected/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-api-access-9sztk\") pod \"error-404-isvc-7cae9-predictor-b8b577bfb-tzf27\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") " pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.059086 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.058980    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7cae9-predictor-b8b577bfb-tzf27\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") " pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.059086 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.059042    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f12ff5e3-1699-4900-a32b-e2da7d13a124-proxy-tls\") pod \"error-404-isvc-7cae9-predictor-b8b577bfb-tzf27\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") " pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.090723 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.090701    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:15.160098 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.159920    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sztk\" (UniqueName: \"kubernetes.io/projected/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-api-access-9sztk\") pod \"error-404-isvc-7cae9-predictor-b8b577bfb-tzf27\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") " pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.160277 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.160256    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7cae9-predictor-b8b577bfb-tzf27\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") " pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.160363 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.160335    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f12ff5e3-1699-4900-a32b-e2da7d13a124-proxy-tls\") pod \"error-404-isvc-7cae9-predictor-b8b577bfb-tzf27\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") " pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.165772 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.161081    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7cae9-predictor-b8b577bfb-tzf27\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") " pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.165772 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.163405    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f12ff5e3-1699-4900-a32b-e2da7d13a124-proxy-tls\") pod \"error-404-isvc-7cae9-predictor-b8b577bfb-tzf27\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") " pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.171718 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.171698    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sztk\" (UniqueName: \"kubernetes.io/projected/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-api-access-9sztk\") pod \"error-404-isvc-7cae9-predictor-b8b577bfb-tzf27\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") " pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.191859 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.191841    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:15.268223 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.268166    2553 generic.go:356] "Generic (PLEG): container finished" podID="d878539b-0f35-4694-879b-349f708a32b5" containerID="eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132" exitCode=2
Jul 28 13:16:15.268305 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.268222    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" event={"ID":"d878539b-0f35-4694-879b-349f708a32b5","Type":"ContainerDied","Data":"eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132"}
Jul 28 13:16:15.273361 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.273337    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:15.396712 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:15.396688    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"]
Jul 28 13:16:15.399009 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:16:15.398982    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12ff5e3_1699_4900_a32b_e2da7d13a124.slice/crio-9dc83c8d7c6f99e393ecccd8b1872c7253c13fe97ba9166aa312616a2e28bb6e WatchSource:0}: Error finding container 9dc83c8d7c6f99e393ecccd8b1872c7253c13fe97ba9166aa312616a2e28bb6e: Status 404 returned error can't find the container with id 9dc83c8d7c6f99e393ecccd8b1872c7253c13fe97ba9166aa312616a2e28bb6e
Jul 28 13:16:16.273871 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:16.273832    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" event={"ID":"f12ff5e3-1699-4900-a32b-e2da7d13a124","Type":"ContainerStarted","Data":"14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a"}
Jul 28 13:16:16.274230 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:16.273897    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" event={"ID":"f12ff5e3-1699-4900-a32b-e2da7d13a124","Type":"ContainerStarted","Data":"3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332"}
Jul 28 13:16:16.274230 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:16.273926    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" event={"ID":"f12ff5e3-1699-4900-a32b-e2da7d13a124","Type":"ContainerStarted","Data":"9dc83c8d7c6f99e393ecccd8b1872c7253c13fe97ba9166aa312616a2e28bb6e"}
Jul 28 13:16:16.274230 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:16.273957    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:16.294478 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:16.294425    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podStartSLOduration=2.294411582 podStartE2EDuration="2.294411582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:16:16.293109899 +0000 UTC m=+1346.330478807" watchObservedRunningTime="2026-07-28 13:16:16.294411582 +0000 UTC m=+1346.331780489"
Jul 28 13:16:16.982574 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:16.982536    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 13:16:17.133618 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.133580    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 13:16:17.280634 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.280547    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:17.281831 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.281785    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:16:17.687587 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.687561    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:16:17.784227 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.784199    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d878539b-0f35-4694-879b-349f708a32b5-proxy-tls\") pod \"d878539b-0f35-4694-879b-349f708a32b5\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") "
Jul 28 13:16:17.784227 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.784231    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl8sg\" (UniqueName: \"kubernetes.io/projected/d878539b-0f35-4694-879b-349f708a32b5-kube-api-access-dl8sg\") pod \"d878539b-0f35-4694-879b-349f708a32b5\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") "
Jul 28 13:16:17.784409 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.784279    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d878539b-0f35-4694-879b-349f708a32b5-kube-rbac-proxy-sar-config\") pod \"d878539b-0f35-4694-879b-349f708a32b5\" (UID: \"d878539b-0f35-4694-879b-349f708a32b5\") "
Jul 28 13:16:17.784612 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.784591    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d878539b-0f35-4694-879b-349f708a32b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d878539b-0f35-4694-879b-349f708a32b5" (UID: "d878539b-0f35-4694-879b-349f708a32b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:16:17.786205 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.786185    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d878539b-0f35-4694-879b-349f708a32b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d878539b-0f35-4694-879b-349f708a32b5" (UID: "d878539b-0f35-4694-879b-349f708a32b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:16:17.786270 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.786253    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d878539b-0f35-4694-879b-349f708a32b5-kube-api-access-dl8sg" (OuterVolumeSpecName: "kube-api-access-dl8sg") pod "d878539b-0f35-4694-879b-349f708a32b5" (UID: "d878539b-0f35-4694-879b-349f708a32b5"). InnerVolumeSpecName "kube-api-access-dl8sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:16:17.885198 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.885138    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d878539b-0f35-4694-879b-349f708a32b5-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:16:17.885198 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.885160    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl8sg\" (UniqueName: \"kubernetes.io/projected/d878539b-0f35-4694-879b-349f708a32b5-kube-api-access-dl8sg\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:16:17.885198 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:17.885169    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d878539b-0f35-4694-879b-349f708a32b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:16:18.285684 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.285649    2553 generic.go:356] "Generic (PLEG): container finished" podID="d878539b-0f35-4694-879b-349f708a32b5" containerID="c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec" exitCode=0
Jul 28 13:16:18.286162 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.285708    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"
Jul 28 13:16:18.286162 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.285732    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" event={"ID":"d878539b-0f35-4694-879b-349f708a32b5","Type":"ContainerDied","Data":"c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec"}
Jul 28 13:16:18.286162 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.285805    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv" event={"ID":"d878539b-0f35-4694-879b-349f708a32b5","Type":"ContainerDied","Data":"c6ec6618b4bbd196d63b346defff9cd2472002794a085ee4f3d618efa5459b82"}
Jul 28 13:16:18.286162 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.285834    2553 scope.go:117] "RemoveContainer" containerID="eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132"
Jul 28 13:16:18.286387 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.286347    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:16:18.294302 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.294288    2553 scope.go:117] "RemoveContainer" containerID="c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec"
Jul 28 13:16:18.302911 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.302893    2553 scope.go:117] "RemoveContainer" containerID="eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132"
Jul 28 13:16:18.303180 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:16:18.303155    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132\": container with ID starting with eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132 not found: ID does not exist" containerID="eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132"
Jul 28 13:16:18.303240 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.303191    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132"} err="failed to get container status \"eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132\": rpc error: code = NotFound desc = could not find container \"eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132\": container with ID starting with eeb4dbeb0e975a61af31b0ec3c6f91e82cdc34a9d88d1b5d4f35bac5701a9132 not found: ID does not exist"
Jul 28 13:16:18.303240 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.303215    2553 scope.go:117] "RemoveContainer" containerID="c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec"
Jul 28 13:16:18.303493 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:16:18.303473    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec\": container with ID starting with c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec not found: ID does not exist" containerID="c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec"
Jul 28 13:16:18.303550 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.303503    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec"} err="failed to get container status \"c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec\": rpc error: code = NotFound desc = could not find container \"c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec\": container with ID starting with c7ca06bc693f4d9a174de4c9a2860d55e922682fb1b3a0b335cade3951d480ec not found: ID does not exist"
Jul 28 13:16:18.307308 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.307287    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"]
Jul 28 13:16:18.311053 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.311034    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dec07-predictor-799fbb6756-wjkbv"]
Jul 28 13:16:18.476253 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:18.476224    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d878539b-0f35-4694-879b-349f708a32b5" path="/var/lib/kubelet/pods/d878539b-0f35-4694-879b-349f708a32b5/volumes"
Jul 28 13:16:23.290878 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:23.290850    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:16:23.291280 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:23.291254    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:16:27.133433 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:27.133406    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:16:33.292119 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:33.292075    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:16:41.895342 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:41.895302    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:43.292191 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:43.292153    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:16:43.821770 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:43.821703    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:50.513146 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.513110    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:50.645971 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.645934    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:50.676706 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.676667    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"]
Jul 28 13:16:50.676999 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.676972    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container" containerID="cri-o://5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409" gracePeriod=30
Jul 28 13:16:50.677082 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.676987    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kube-rbac-proxy" containerID="cri-o://479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5" gracePeriod=30
Jul 28 13:16:50.718163 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.718063    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:50.787447 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.787417    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"]
Jul 28 13:16:50.787829 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.787814    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kube-rbac-proxy"
Jul 28 13:16:50.787878 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.787831    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kube-rbac-proxy"
Jul 28 13:16:50.787878 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.787841    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container"
Jul 28 13:16:50.787878 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.787848    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container"
Jul 28 13:16:50.787974 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.787902    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kube-rbac-proxy"
Jul 28 13:16:50.787974 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.787915    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d878539b-0f35-4694-879b-349f708a32b5" containerName="kserve-container"
Jul 28 13:16:50.789585 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.789561    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:50.792318 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.792301    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:50.795319 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.795297    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dd548-kube-rbac-proxy-sar-config\""
Jul 28 13:16:50.795428 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.795306    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dd548-predictor-serving-cert\""
Jul 28 13:16:50.802623 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.802600    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"]
Jul 28 13:16:50.845654 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.845617    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dd548-predictor-5877b95dd-26m7l\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") " pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:50.845871 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.845667    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75w4j\" (UniqueName: \"kubernetes.io/projected/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-api-access-75w4j\") pod \"error-404-isvc-dd548-predictor-5877b95dd-26m7l\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") " pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:50.845871 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.845789    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-proxy-tls\") pod \"error-404-isvc-dd548-predictor-5877b95dd-26m7l\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") " pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:50.854396 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.854364    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:50.946564 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.946527    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dd548-predictor-5877b95dd-26m7l\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") " pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:50.946564 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.946575    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75w4j\" (UniqueName: \"kubernetes.io/projected/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-api-access-75w4j\") pod \"error-404-isvc-dd548-predictor-5877b95dd-26m7l\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") " pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:50.946845 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.946626    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-proxy-tls\") pod \"error-404-isvc-dd548-predictor-5877b95dd-26m7l\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") " pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:50.947203 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.947181    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dd548-predictor-5877b95dd-26m7l\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") " pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:50.949005 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.948987    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-proxy-tls\") pod \"error-404-isvc-dd548-predictor-5877b95dd-26m7l\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") " pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:50.954290 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.954268    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:16:50.955265 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:50.955242    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75w4j\" (UniqueName: \"kubernetes.io/projected/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-api-access-75w4j\") pod \"error-404-isvc-dd548-predictor-5877b95dd-26m7l\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") " pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:51.104211 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.104096    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:51.239182 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.239154    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"]
Jul 28 13:16:51.241112 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:16:51.241083    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d21f1a_d1c8_49fb_91ff_89ae6f29b28c.slice/crio-45ccf8042c93ce0162eb1a66878ed5f6caa1911f694563618b8746a4168ba92d WatchSource:0}: Error finding container 45ccf8042c93ce0162eb1a66878ed5f6caa1911f694563618b8746a4168ba92d: Status 404 returned error can't find the container with id 45ccf8042c93ce0162eb1a66878ed5f6caa1911f694563618b8746a4168ba92d
Jul 28 13:16:51.430239 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.430205    2553 generic.go:356] "Generic (PLEG): container finished" podID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerID="479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5" exitCode=2
Jul 28 13:16:51.430435 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.430279    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" event={"ID":"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28","Type":"ContainerDied","Data":"479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5"}
Jul 28 13:16:51.431993 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.431969    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" event={"ID":"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c","Type":"ContainerStarted","Data":"7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924"}
Jul 28 13:16:51.432150 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.432011    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" event={"ID":"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c","Type":"ContainerStarted","Data":"302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827"}
Jul 28 13:16:51.432150 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.432029    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" event={"ID":"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c","Type":"ContainerStarted","Data":"45ccf8042c93ce0162eb1a66878ed5f6caa1911f694563618b8746a4168ba92d"}
Jul 28 13:16:51.432260 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.432237    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:51.432313 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.432269    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:51.433657 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.433633    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:16:51.450997 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:51.450954    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podStartSLOduration=1.450942109 podStartE2EDuration="1.450942109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:16:51.450289857 +0000 UTC m=+1381.487658770" watchObservedRunningTime="2026-07-28 13:16:51.450942109 +0000 UTC m=+1381.488311017"
Jul 28 13:16:52.128687 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:52.128634    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 13:16:52.436495 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:52.436454    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:16:53.292181 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:53.292139    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:16:54.329228 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.329203    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:16:54.378150 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.378074    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-proxy-tls\") pod \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") "
Jul 28 13:16:54.378150 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.378119    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztg8w\" (UniqueName: \"kubernetes.io/projected/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-api-access-ztg8w\") pod \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") "
Jul 28 13:16:54.378377 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.378166    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-rbac-proxy-sar-config\") pod \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\" (UID: \"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28\") "
Jul 28 13:16:54.378551 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.378528    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" (UID: "e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:16:54.380219 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.380197    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" (UID: "e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:16:54.380300 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.380266    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-api-access-ztg8w" (OuterVolumeSpecName: "kube-api-access-ztg8w") pod "e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" (UID: "e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28"). InnerVolumeSpecName "kube-api-access-ztg8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:16:54.447334 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.447300    2553 generic.go:356] "Generic (PLEG): container finished" podID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerID="5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409" exitCode=0
Jul 28 13:16:54.447486 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.447355    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"
Jul 28 13:16:54.447486 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.447381    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" event={"ID":"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28","Type":"ContainerDied","Data":"5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409"}
Jul 28 13:16:54.447486 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.447441    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl" event={"ID":"e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28","Type":"ContainerDied","Data":"e758c8c99b7ebb5e4378f8e0205c8127750ebdb0f3039a0165de3e2fa7c4a5fe"}
Jul 28 13:16:54.447486 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.447468    2553 scope.go:117] "RemoveContainer" containerID="479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5"
Jul 28 13:16:54.456105 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.456087    2553 scope.go:117] "RemoveContainer" containerID="5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409"
Jul 28 13:16:54.463581 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.463564    2553 scope.go:117] "RemoveContainer" containerID="479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5"
Jul 28 13:16:54.463929 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:16:54.463911    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5\": container with ID starting with 479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5 not found: ID does not exist" containerID="479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5"
Jul 28 13:16:54.463990 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.463939    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5"} err="failed to get container status \"479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5\": rpc error: code = NotFound desc = could not find container \"479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5\": container with ID starting with 479418fc5b419449eeec184388354f14c7d2b74061e0c735985a946bc79baee5 not found: ID does not exist"
Jul 28 13:16:54.463990 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.463958    2553 scope.go:117] "RemoveContainer" containerID="5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409"
Jul 28 13:16:54.464208 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:16:54.464190    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409\": container with ID starting with 5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409 not found: ID does not exist" containerID="5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409"
Jul 28 13:16:54.464253 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.464213    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409"} err="failed to get container status \"5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409\": rpc error: code = NotFound desc = could not find container \"5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409\": container with ID starting with 5e65a1ccfee45dfc0487da804b6905ffba6dba4521784cb24ba2e75f5d4dd409 not found: ID does not exist"
Jul 28 13:16:54.470760 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.470738    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"]
Jul 28 13:16:54.477900 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.477882    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ff2e-predictor-7484d48446-grrpl"]
Jul 28 13:16:54.478827 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.478813    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztg8w\" (UniqueName: \"kubernetes.io/projected/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-api-access-ztg8w\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:16:54.478880 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.478829    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:16:54.478880 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:54.478840    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:16:56.476144 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:56.476110    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" path="/var/lib/kubelet/pods/e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28/volumes"
Jul 28 13:16:57.441150 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:57.441122    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:16:57.441599 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:16:57.441571    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:17:03.291941 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:17:03.291907    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:17:07.441544 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:17:07.441507    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:17:14.934521 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:17:14.934488    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:17:15.172313 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:17:15.172281    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:17:17.441858 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:17:17.441789    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:17:27.442429 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:17:27.442387    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:17:37.441970 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:17:37.441935    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:17:50.795842 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:17:50.795721    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:17:51.268144 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:17:51.268109    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:18:06.386270 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:18:06.386233    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:19:00.883896 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:19:00.883855    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:19:38.801916 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:19:38.801874    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:19:38.930035 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:19:38.929996    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:20:26.315156 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:20:26.315115    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:25:29.568024 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.567922    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:25:29.827850 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.827746    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"]
Jul 28 13:25:29.828140 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.828128    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kube-rbac-proxy"
Jul 28 13:25:29.828184 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.828142    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kube-rbac-proxy"
Jul 28 13:25:29.828184 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.828164    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container"
Jul 28 13:25:29.828184 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.828170    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container"
Jul 28 13:25:29.828277 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.828224    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kserve-container"
Jul 28 13:25:29.828277 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.828234    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6d4ffd3-1013-4d9f-ac54-bc8b5fdf8f28" containerName="kube-rbac-proxy"
Jul 28 13:25:29.831331 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.831311    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:29.835446 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.835424    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-198f4-predictor-serving-cert\""
Jul 28 13:25:29.835695 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.835674    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-198f4-kube-rbac-proxy-sar-config\""
Jul 28 13:25:29.841559 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.841529    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"]
Jul 28 13:25:29.896064 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.896026    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"]
Jul 28 13:25:29.896374 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.896326    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container" containerID="cri-o://3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332" gracePeriod=30
Jul 28 13:25:29.896472 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.896370    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kube-rbac-proxy" containerID="cri-o://14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a" gracePeriod=30
Jul 28 13:25:29.935178 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.935140    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dcd5827-68af-4199-93d3-1729343138ff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:29.935362 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.935203    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8zs\" (UniqueName: \"kubernetes.io/projected/6dcd5827-68af-4199-93d3-1729343138ff-kube-api-access-fk8zs\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:29.935362 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:29.935303    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dcd5827-68af-4199-93d3-1729343138ff-proxy-tls\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:30.036229 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.036186    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dcd5827-68af-4199-93d3-1729343138ff-proxy-tls\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:30.036384 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.036251    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dcd5827-68af-4199-93d3-1729343138ff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:30.036384 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.036300    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8zs\" (UniqueName: \"kubernetes.io/projected/6dcd5827-68af-4199-93d3-1729343138ff-kube-api-access-fk8zs\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:30.036384 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:25:30.036362    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-198f4-predictor-serving-cert: secret "error-404-isvc-198f4-predictor-serving-cert" not found
Jul 28 13:25:30.036520 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:25:30.036480    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dcd5827-68af-4199-93d3-1729343138ff-proxy-tls podName:6dcd5827-68af-4199-93d3-1729343138ff nodeName:}" failed. No retries permitted until 2026-07-28 13:25:30.536451482 +0000 UTC m=+1900.573820375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6dcd5827-68af-4199-93d3-1729343138ff-proxy-tls") pod "error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" (UID: "6dcd5827-68af-4199-93d3-1729343138ff") : secret "error-404-isvc-198f4-predictor-serving-cert" not found
Jul 28 13:25:30.036914 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.036895    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dcd5827-68af-4199-93d3-1729343138ff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:30.046750 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.046715    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8zs\" (UniqueName: \"kubernetes.io/projected/6dcd5827-68af-4199-93d3-1729343138ff-kube-api-access-fk8zs\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:30.541131 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.541080    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dcd5827-68af-4199-93d3-1729343138ff-proxy-tls\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:30.543553 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.543529    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dcd5827-68af-4199-93d3-1729343138ff-proxy-tls\") pod \"error-404-isvc-198f4-predictor-f9dd4f67d-5glr6\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") " pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:30.632371 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.632336    2553 generic.go:356] "Generic (PLEG): container finished" podID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerID="14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a" exitCode=2
Jul 28 13:25:30.632771 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.632419    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" event={"ID":"f12ff5e3-1699-4900-a32b-e2da7d13a124","Type":"ContainerDied","Data":"14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a"}
Jul 28 13:25:30.682296 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.682246    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:25:30.742777 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.742729    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:30.882187 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:30.882159    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"]
Jul 28 13:25:30.884738 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:25:30.884707    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcd5827_68af_4199_93d3_1729343138ff.slice/crio-5bed32e4ef82ed113abafa4e8176659f6a69380bf67c5e3ebefed303f16576ee WatchSource:0}: Error finding container 5bed32e4ef82ed113abafa4e8176659f6a69380bf67c5e3ebefed303f16576ee: Status 404 returned error can't find the container with id 5bed32e4ef82ed113abafa4e8176659f6a69380bf67c5e3ebefed303f16576ee
Jul 28 13:25:31.638494 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:31.638454    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" event={"ID":"6dcd5827-68af-4199-93d3-1729343138ff","Type":"ContainerStarted","Data":"fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab"}
Jul 28 13:25:31.638940 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:31.638512    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" event={"ID":"6dcd5827-68af-4199-93d3-1729343138ff","Type":"ContainerStarted","Data":"7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9"}
Jul 28 13:25:31.638940 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:31.638535    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" event={"ID":"6dcd5827-68af-4199-93d3-1729343138ff","Type":"ContainerStarted","Data":"5bed32e4ef82ed113abafa4e8176659f6a69380bf67c5e3ebefed303f16576ee"}
Jul 28 13:25:31.638940 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:31.638613    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:31.663258 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:31.663210    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" podStartSLOduration=2.663193007 podStartE2EDuration="2.663193007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:25:31.660993953 +0000 UTC m=+1901.698362860" watchObservedRunningTime="2026-07-28 13:25:31.663193007 +0000 UTC m=+1901.700561915"
Jul 28 13:25:32.643143 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:32.643111    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:32.644323 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:32.644297    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:25:33.287287 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.287235    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 13:25:33.291537 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.291513    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:25:33.449007 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.448977    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:25:33.567632 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.567530    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f12ff5e3-1699-4900-a32b-e2da7d13a124-proxy-tls\") pod \"f12ff5e3-1699-4900-a32b-e2da7d13a124\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") "
Jul 28 13:25:33.567632 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.567578    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sztk\" (UniqueName: \"kubernetes.io/projected/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-api-access-9sztk\") pod \"f12ff5e3-1699-4900-a32b-e2da7d13a124\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") "
Jul 28 13:25:33.567632 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.567633    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-rbac-proxy-sar-config\") pod \"f12ff5e3-1699-4900-a32b-e2da7d13a124\" (UID: \"f12ff5e3-1699-4900-a32b-e2da7d13a124\") "
Jul 28 13:25:33.568095 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.568071    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f12ff5e3-1699-4900-a32b-e2da7d13a124" (UID: "f12ff5e3-1699-4900-a32b-e2da7d13a124"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:25:33.569789 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.569762    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-api-access-9sztk" (OuterVolumeSpecName: "kube-api-access-9sztk") pod "f12ff5e3-1699-4900-a32b-e2da7d13a124" (UID: "f12ff5e3-1699-4900-a32b-e2da7d13a124"). InnerVolumeSpecName "kube-api-access-9sztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:25:33.569789 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.569779    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12ff5e3-1699-4900-a32b-e2da7d13a124-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f12ff5e3-1699-4900-a32b-e2da7d13a124" (UID: "f12ff5e3-1699-4900-a32b-e2da7d13a124"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:25:33.649301 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.649263    2553 generic.go:356] "Generic (PLEG): container finished" podID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerID="3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332" exitCode=0
Jul 28 13:25:33.649746 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.649316    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"
Jul 28 13:25:33.649746 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.649349    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" event={"ID":"f12ff5e3-1699-4900-a32b-e2da7d13a124","Type":"ContainerDied","Data":"3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332"}
Jul 28 13:25:33.649746 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.649454    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27" event={"ID":"f12ff5e3-1699-4900-a32b-e2da7d13a124","Type":"ContainerDied","Data":"9dc83c8d7c6f99e393ecccd8b1872c7253c13fe97ba9166aa312616a2e28bb6e"}
Jul 28 13:25:33.649746 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.649489    2553 scope.go:117] "RemoveContainer" containerID="14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a"
Jul 28 13:25:33.650052 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.650028    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:25:33.658695 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.658674    2553 scope.go:117] "RemoveContainer" containerID="3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332"
Jul 28 13:25:33.666618 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.666600    2553 scope.go:117] "RemoveContainer" containerID="14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a"
Jul 28 13:25:33.666960 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:25:33.666939    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a\": container with ID starting with 14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a not found: ID does not exist" containerID="14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a"
Jul 28 13:25:33.667032 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.666973    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a"} err="failed to get container status \"14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a\": rpc error: code = NotFound desc = could not find container \"14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a\": container with ID starting with 14f9b1cd8e3c745beaa4903b9c7bebab604e64324e3e0accdfa38ffa6037cd8a not found: ID does not exist"
Jul 28 13:25:33.667032 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.667001    2553 scope.go:117] "RemoveContainer" containerID="3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332"
Jul 28 13:25:33.667267 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:25:33.667251    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332\": container with ID starting with 3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332 not found: ID does not exist" containerID="3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332"
Jul 28 13:25:33.667317 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.667276    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332"} err="failed to get container status \"3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332\": rpc error: code = NotFound desc = could not find container \"3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332\": container with ID starting with 3aab9a45e18e48fde919b28f9d01b9839b06d425b2e253eba721b4ea58c14332 not found: ID does not exist"
Jul 28 13:25:33.668691 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.668672    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f12ff5e3-1699-4900-a32b-e2da7d13a124-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:25:33.668738 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.668695    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sztk\" (UniqueName: \"kubernetes.io/projected/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-api-access-9sztk\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:25:33.668738 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.668705    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f12ff5e3-1699-4900-a32b-e2da7d13a124-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:25:33.672045 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.672020    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"]
Jul 28 13:25:33.677217 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:33.677181    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cae9-predictor-b8b577bfb-tzf27"]
Jul 28 13:25:34.475961 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:34.475930    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" path="/var/lib/kubelet/pods/f12ff5e3-1699-4900-a32b-e2da7d13a124/volumes"
Jul 28 13:25:38.654404 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:38.654372    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:25:38.654896 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:38.654777    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:25:48.654867 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:48.654825    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:25:58.654991 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:25:58.654944    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:26:05.381951 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.381914    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:05.501440 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.501401    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:05.533396 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.533360    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"]
Jul 28 13:26:05.533622 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.533594    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container" containerID="cri-o://302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827" gracePeriod=30
Jul 28 13:26:05.533777 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.533649    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kube-rbac-proxy" containerID="cri-o://7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924" gracePeriod=30
Jul 28 13:26:05.574706 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.574676    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:05.686485 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.686448    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"]
Jul 28 13:26:05.686846 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.686832    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kube-rbac-proxy"
Jul 28 13:26:05.686902 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.686848    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kube-rbac-proxy"
Jul 28 13:26:05.686902 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.686866    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container"
Jul 28 13:26:05.686902 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.686872    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container"
Jul 28 13:26:05.686998 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.686951    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kserve-container"
Jul 28 13:26:05.686998 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.686962    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12ff5e3-1699-4900-a32b-e2da7d13a124" containerName="kube-rbac-proxy"
Jul 28 13:26:05.690012 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.689995    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.692902 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.692878    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-18d3f-predictor-serving-cert\""
Jul 28 13:26:05.693068 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.692889    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-18d3f-kube-rbac-proxy-sar-config\""
Jul 28 13:26:05.699549 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.699524    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"]
Jul 28 13:26:05.742690 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.742658    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:05.763450 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.763407    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316b03dc-65ce-4d8d-8f1f-3d44a049a440-proxy-tls\") pod \"error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") " pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.763676 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.763543    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") " pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.763676 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.763583    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87txm\" (UniqueName: \"kubernetes.io/projected/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-api-access-87txm\") pod \"error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") " pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.786857 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.786822    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerID="7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924" exitCode=2
Jul 28 13:26:05.787008 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.786885    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" event={"ID":"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c","Type":"ContainerDied","Data":"7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924"}
Jul 28 13:26:05.864285 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.864246    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316b03dc-65ce-4d8d-8f1f-3d44a049a440-proxy-tls\") pod \"error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") " pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.864474 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.864329    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") " pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.864474 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.864347    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87txm\" (UniqueName: \"kubernetes.io/projected/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-api-access-87txm\") pod \"error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") " pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.865105 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.865085    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") " pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.866757 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.866740    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316b03dc-65ce-4d8d-8f1f-3d44a049a440-proxy-tls\") pod \"error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") " pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.873424 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.873402    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87txm\" (UniqueName: \"kubernetes.io/projected/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-api-access-87txm\") pod \"error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") " pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:05.888344 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:05.888319    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:06.001672 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.001636    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:06.108243 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.108176    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:06.141610 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.141584    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"]
Jul 28 13:26:06.143573 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:26:06.143547    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316b03dc_65ce_4d8d_8f1f_3d44a049a440.slice/crio-8bfde4628ee0a04e50e974ff2fcbb6597356459808056a12a3375c39045d2933 WatchSource:0}: Error finding container 8bfde4628ee0a04e50e974ff2fcbb6597356459808056a12a3375c39045d2933: Status 404 returned error can't find the container with id 8bfde4628ee0a04e50e974ff2fcbb6597356459808056a12a3375c39045d2933
Jul 28 13:26:06.792899 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.792860    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" event={"ID":"316b03dc-65ce-4d8d-8f1f-3d44a049a440","Type":"ContainerStarted","Data":"5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d"}
Jul 28 13:26:06.793346 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.792921    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" event={"ID":"316b03dc-65ce-4d8d-8f1f-3d44a049a440","Type":"ContainerStarted","Data":"daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e"}
Jul 28 13:26:06.793346 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.792941    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" event={"ID":"316b03dc-65ce-4d8d-8f1f-3d44a049a440","Type":"ContainerStarted","Data":"8bfde4628ee0a04e50e974ff2fcbb6597356459808056a12a3375c39045d2933"}
Jul 28 13:26:06.793346 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.793124    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:06.793346 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.793279    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:06.794439 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.794410    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 13:26:06.813434 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:06.813365    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podStartSLOduration=1.8133487160000001 podStartE2EDuration="1.813348716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:26:06.811223357 +0000 UTC m=+1936.848592294" watchObservedRunningTime="2026-07-28 13:26:06.813348716 +0000 UTC m=+1936.850717624"
Jul 28 13:26:07.437323 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:07.437275    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 13:26:07.441585 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:07.441561    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:26:07.797213 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:07.797106    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 13:26:08.655249 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:08.655212    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:26:09.307651 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.307617    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:26:09.391504 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.391412    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75w4j\" (UniqueName: \"kubernetes.io/projected/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-api-access-75w4j\") pod \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") "
Jul 28 13:26:09.391653 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.391523    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-proxy-tls\") pod \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") "
Jul 28 13:26:09.391653 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.391552    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-rbac-proxy-sar-config\") pod \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\" (UID: \"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c\") "
Jul 28 13:26:09.392002 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.391970    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" (UID: "b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:26:09.393706 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.393679    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" (UID: "b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:26:09.393811 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.393722    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-api-access-75w4j" (OuterVolumeSpecName: "kube-api-access-75w4j") pod "b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" (UID: "b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c"). InnerVolumeSpecName "kube-api-access-75w4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:26:09.492674 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.492637    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:26:09.492674 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.492668    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:26:09.492674 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.492677    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75w4j\" (UniqueName: \"kubernetes.io/projected/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c-kube-api-access-75w4j\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:26:09.809695 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.809661    2553 generic.go:356] "Generic (PLEG): container finished" podID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerID="302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827" exitCode=0
Jul 28 13:26:09.810031 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.809706    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"
Jul 28 13:26:09.810031 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.809705    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" event={"ID":"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c","Type":"ContainerDied","Data":"302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827"}
Jul 28 13:26:09.810031 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.809840    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l" event={"ID":"b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c","Type":"ContainerDied","Data":"45ccf8042c93ce0162eb1a66878ed5f6caa1911f694563618b8746a4168ba92d"}
Jul 28 13:26:09.810031 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.809872    2553 scope.go:117] "RemoveContainer" containerID="7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924"
Jul 28 13:26:09.819114 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.819100    2553 scope.go:117] "RemoveContainer" containerID="302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827"
Jul 28 13:26:09.826602 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.826583    2553 scope.go:117] "RemoveContainer" containerID="7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924"
Jul 28 13:26:09.826888 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:26:09.826867    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924\": container with ID starting with 7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924 not found: ID does not exist" containerID="7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924"
Jul 28 13:26:09.826971 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.826899    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924"} err="failed to get container status \"7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924\": rpc error: code = NotFound desc = could not find container \"7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924\": container with ID starting with 7cece2b2ebb895fd5b2a85607b9e2261deaf8e3ec676e01dc3930d80c1cc7924 not found: ID does not exist"
Jul 28 13:26:09.826971 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.826924    2553 scope.go:117] "RemoveContainer" containerID="302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827"
Jul 28 13:26:09.827169 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:26:09.827153    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827\": container with ID starting with 302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827 not found: ID does not exist" containerID="302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827"
Jul 28 13:26:09.827245 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.827177    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827"} err="failed to get container status \"302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827\": rpc error: code = NotFound desc = could not find container \"302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827\": container with ID starting with 302bb70910acd4314f2034171e70fef9f7384633de9c12a198362f0363166827 not found: ID does not exist"
Jul 28 13:26:09.833245 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.833223    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"]
Jul 28 13:26:09.839206 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:09.839185    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dd548-predictor-5877b95dd-26m7l"]
Jul 28 13:26:10.476419 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:10.476386    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" path="/var/lib/kubelet/pods/b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c/volumes"
Jul 28 13:26:12.804242 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:12.804209    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:12.804897 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:12.804863    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 13:26:18.655507 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:18.655476    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:26:22.805513 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:22.805457    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 13:26:29.753338 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:29.753303    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:30.149244 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:30.149148    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:32.805628 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:32.805579    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 13:26:39.851590 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:39.851506    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:39.966213 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:39.966177    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:40.007486 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.007445    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"]
Jul 28 13:26:40.007732 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.007705    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container" containerID="cri-o://7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9" gracePeriod=30
Jul 28 13:26:40.007828 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.007718    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kube-rbac-proxy" containerID="cri-o://fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab" gracePeriod=30
Jul 28 13:26:40.120170 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.120089    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"]
Jul 28 13:26:40.120632 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.120611    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container"
Jul 28 13:26:40.120632 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.120631    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container"
Jul 28 13:26:40.120774 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.120652    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kube-rbac-proxy"
Jul 28 13:26:40.120774 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.120657    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kube-rbac-proxy"
Jul 28 13:26:40.120774 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.120722    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kserve-container"
Jul 28 13:26:40.120774 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.120734    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d21f1a-d1c8-49fb-91ff-89ae6f29b28c" containerName="kube-rbac-proxy"
Jul 28 13:26:40.124116 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.124098    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.126769 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.126749    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b971b-kube-rbac-proxy-sar-config\""
Jul 28 13:26:40.126857 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.126751    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b971b-predictor-serving-cert\""
Jul 28 13:26:40.128847 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.128807    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:40.133311 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.133284    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"]
Jul 28 13:26:40.226130 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.226067    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:40.265190 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.265151    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82c22232-ba57-477d-812f-81bb8879c5dc-proxy-tls\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.265374 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.265209    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txzz\" (UniqueName: \"kubernetes.io/projected/82c22232-ba57-477d-812f-81bb8879c5dc-kube-api-access-8txzz\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.265374 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.265266    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82c22232-ba57-477d-812f-81bb8879c5dc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.365967 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.365924    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82c22232-ba57-477d-812f-81bb8879c5dc-proxy-tls\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.366149 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.365988    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txzz\" (UniqueName: \"kubernetes.io/projected/82c22232-ba57-477d-812f-81bb8879c5dc-kube-api-access-8txzz\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.366149 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.366018    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82c22232-ba57-477d-812f-81bb8879c5dc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.366149 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:26:40.366078    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b971b-predictor-serving-cert: secret "error-404-isvc-b971b-predictor-serving-cert" not found
Jul 28 13:26:40.366149 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:26:40.366148    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82c22232-ba57-477d-812f-81bb8879c5dc-proxy-tls podName:82c22232-ba57-477d-812f-81bb8879c5dc nodeName:}" failed. No retries permitted until 2026-07-28 13:26:40.866129158 +0000 UTC m=+1970.903498047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/82c22232-ba57-477d-812f-81bb8879c5dc-proxy-tls") pod "error-404-isvc-b971b-predictor-75f98f6896-m2rmj" (UID: "82c22232-ba57-477d-812f-81bb8879c5dc") : secret "error-404-isvc-b971b-predictor-serving-cert" not found
Jul 28 13:26:40.366748 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.366731    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82c22232-ba57-477d-812f-81bb8879c5dc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.375942 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.375880    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txzz\" (UniqueName: \"kubernetes.io/projected/82c22232-ba57-477d-812f-81bb8879c5dc-kube-api-access-8txzz\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.453548 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.453510    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:26:40.871518 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.871472    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82c22232-ba57-477d-812f-81bb8879c5dc-proxy-tls\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.873975 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.873953    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82c22232-ba57-477d-812f-81bb8879c5dc-proxy-tls\") pod \"error-404-isvc-b971b-predictor-75f98f6896-m2rmj\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") " pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:40.946481 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.946440    2553 generic.go:356] "Generic (PLEG): container finished" podID="6dcd5827-68af-4199-93d3-1729343138ff" containerID="fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab" exitCode=2
Jul 28 13:26:40.946657 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:40.946503    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" event={"ID":"6dcd5827-68af-4199-93d3-1729343138ff","Type":"ContainerDied","Data":"fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab"}
Jul 28 13:26:41.035817 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:41.035747    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:41.171830 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:41.171786    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"]
Jul 28 13:26:41.174028 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:26:41.174001    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c22232_ba57_477d_812f_81bb8879c5dc.slice/crio-a3a8c8025ce641ea223d3108712396aece28aa2c58f4642a141a19d979878a1b WatchSource:0}: Error finding container a3a8c8025ce641ea223d3108712396aece28aa2c58f4642a141a19d979878a1b: Status 404 returned error can't find the container with id a3a8c8025ce641ea223d3108712396aece28aa2c58f4642a141a19d979878a1b
Jul 28 13:26:41.952956 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:41.952899    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" event={"ID":"82c22232-ba57-477d-812f-81bb8879c5dc","Type":"ContainerStarted","Data":"9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1"}
Jul 28 13:26:41.953402 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:41.952990    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" event={"ID":"82c22232-ba57-477d-812f-81bb8879c5dc","Type":"ContainerStarted","Data":"a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624"}
Jul 28 13:26:41.953402 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:41.953020    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" event={"ID":"82c22232-ba57-477d-812f-81bb8879c5dc","Type":"ContainerStarted","Data":"a3a8c8025ce641ea223d3108712396aece28aa2c58f4642a141a19d979878a1b"}
Jul 28 13:26:41.953402 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:41.953052    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:41.976508 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:41.976453    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" podStartSLOduration=1.976436361 podStartE2EDuration="1.976436361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:26:41.976037029 +0000 UTC m=+1972.013405937" watchObservedRunningTime="2026-07-28 13:26:41.976436361 +0000 UTC m=+1972.013805268"
Jul 28 13:26:42.804993 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:42.804949    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 13:26:42.956881 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:42.956841    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:42.958225 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:42.958194    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 13:26:43.465880 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.465851    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:26:43.490263 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.490234    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dcd5827-68af-4199-93d3-1729343138ff-proxy-tls\") pod \"6dcd5827-68af-4199-93d3-1729343138ff\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") "
Jul 28 13:26:43.490423 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.490279    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk8zs\" (UniqueName: \"kubernetes.io/projected/6dcd5827-68af-4199-93d3-1729343138ff-kube-api-access-fk8zs\") pod \"6dcd5827-68af-4199-93d3-1729343138ff\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") "
Jul 28 13:26:43.490423 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.490311    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dcd5827-68af-4199-93d3-1729343138ff-kube-rbac-proxy-sar-config\") pod \"6dcd5827-68af-4199-93d3-1729343138ff\" (UID: \"6dcd5827-68af-4199-93d3-1729343138ff\") "
Jul 28 13:26:43.490851 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.490741    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dcd5827-68af-4199-93d3-1729343138ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6dcd5827-68af-4199-93d3-1729343138ff" (UID: "6dcd5827-68af-4199-93d3-1729343138ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:26:43.490986 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.490892    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dcd5827-68af-4199-93d3-1729343138ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:26:43.492515 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.492491    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dcd5827-68af-4199-93d3-1729343138ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6dcd5827-68af-4199-93d3-1729343138ff" (UID: "6dcd5827-68af-4199-93d3-1729343138ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:26:43.492641 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.492522    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcd5827-68af-4199-93d3-1729343138ff-kube-api-access-fk8zs" (OuterVolumeSpecName: "kube-api-access-fk8zs") pod "6dcd5827-68af-4199-93d3-1729343138ff" (UID: "6dcd5827-68af-4199-93d3-1729343138ff"). InnerVolumeSpecName "kube-api-access-fk8zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:26:43.592119 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.592027    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dcd5827-68af-4199-93d3-1729343138ff-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:26:43.592119 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.592060    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8zs\" (UniqueName: \"kubernetes.io/projected/6dcd5827-68af-4199-93d3-1729343138ff-kube-api-access-fk8zs\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:26:43.962633 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.962597    2553 generic.go:356] "Generic (PLEG): container finished" podID="6dcd5827-68af-4199-93d3-1729343138ff" containerID="7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9" exitCode=0
Jul 28 13:26:43.963145 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.962644    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"
Jul 28 13:26:43.963145 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.962682    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" event={"ID":"6dcd5827-68af-4199-93d3-1729343138ff","Type":"ContainerDied","Data":"7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9"}
Jul 28 13:26:43.963145 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.962743    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6" event={"ID":"6dcd5827-68af-4199-93d3-1729343138ff","Type":"ContainerDied","Data":"5bed32e4ef82ed113abafa4e8176659f6a69380bf67c5e3ebefed303f16576ee"}
Jul 28 13:26:43.963145 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.962779    2553 scope.go:117] "RemoveContainer" containerID="fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab"
Jul 28 13:26:43.963433 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.963408    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 13:26:43.971304 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.971277    2553 scope.go:117] "RemoveContainer" containerID="7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9"
Jul 28 13:26:43.979427 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.979408    2553 scope.go:117] "RemoveContainer" containerID="fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab"
Jul 28 13:26:43.979679 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:26:43.979658    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab\": container with ID starting with fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab not found: ID does not exist" containerID="fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab"
Jul 28 13:26:43.979732 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.979688    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab"} err="failed to get container status \"fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab\": rpc error: code = NotFound desc = could not find container \"fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab\": container with ID starting with fe6dd9b602ce8c2e80e28cc2e013091c610682f6368200ecc2be729a6a4813ab not found: ID does not exist"
Jul 28 13:26:43.979732 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.979705    2553 scope.go:117] "RemoveContainer" containerID="7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9"
Jul 28 13:26:43.979957 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:26:43.979938    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9\": container with ID starting with 7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9 not found: ID does not exist" containerID="7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9"
Jul 28 13:26:43.980022 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.979968    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9"} err="failed to get container status \"7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9\": rpc error: code = NotFound desc = could not find container \"7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9\": container with ID starting with 7c80615d46b89f52eb0debb65f77132503cb0a18f2458b441d0f89be781d3be9 not found: ID does not exist"
Jul 28 13:26:43.986100 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.986075    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"]
Jul 28 13:26:43.988522 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:43.988498    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-198f4-predictor-f9dd4f67d-5glr6"]
Jul 28 13:26:44.476563 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:44.476530    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dcd5827-68af-4199-93d3-1729343138ff" path="/var/lib/kubelet/pods/6dcd5827-68af-4199-93d3-1729343138ff/volumes"
Jul 28 13:26:48.968170 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:48.968138    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:26:48.968644 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:48.968615    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 13:26:52.805399 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:52.805362    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:26:58.973479 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:26:58.973429    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 13:27:05.635258 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:27:05.635213    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:27:06.185325 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:27:06.185292    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:27:08.968666 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:27:08.968626    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 13:27:18.968768 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:27:18.968723    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 13:27:28.969282 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:27:28.969241    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:27:40.108164 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:27:40.108122    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:27:40.538550 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:27:40.538512    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:27:40.847896 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:27:40.847779    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:28:02.360723 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:28:02.360682    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:31:33.363614 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:31:33.363522    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:35:02.482212 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:02.482170    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:35:23.933748 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:23.933697    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:35:54.776324 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:54.776236    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:35:54.925677 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:54.925638    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:35:54.945781 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:54.945747    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"]
Jul 28 13:35:54.946072 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:54.946047    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container" containerID="cri-o://a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624" gracePeriod=30
Jul 28 13:35:54.946166 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:54.946138    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kube-rbac-proxy" containerID="cri-o://9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1" gracePeriod=30
Jul 28 13:35:55.007040 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:55.007004    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:35:55.313961 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:55.313875    2553 generic.go:356] "Generic (PLEG): container finished" podID="82c22232-ba57-477d-812f-81bb8879c5dc" containerID="9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1" exitCode=2
Jul 28 13:35:55.313961 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:55.313926    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" event={"ID":"82c22232-ba57-477d-812f-81bb8879c5dc","Type":"ContainerDied","Data":"9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1"}
Jul 28 13:35:58.297731 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.297700    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:35:58.328097 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.328062    2553 generic.go:356] "Generic (PLEG): container finished" podID="82c22232-ba57-477d-812f-81bb8879c5dc" containerID="a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624" exitCode=0
Jul 28 13:35:58.328287 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.328119    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"
Jul 28 13:35:58.328287 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.328150    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" event={"ID":"82c22232-ba57-477d-812f-81bb8879c5dc","Type":"ContainerDied","Data":"a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624"}
Jul 28 13:35:58.328287 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.328206    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj" event={"ID":"82c22232-ba57-477d-812f-81bb8879c5dc","Type":"ContainerDied","Data":"a3a8c8025ce641ea223d3108712396aece28aa2c58f4642a141a19d979878a1b"}
Jul 28 13:35:58.328287 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.328236    2553 scope.go:117] "RemoveContainer" containerID="9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1"
Jul 28 13:35:58.336350 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.336330    2553 scope.go:117] "RemoveContainer" containerID="a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624"
Jul 28 13:35:58.344035 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.344016    2553 scope.go:117] "RemoveContainer" containerID="9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1"
Jul 28 13:35:58.344292 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:35:58.344276    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1\": container with ID starting with 9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1 not found: ID does not exist" containerID="9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1"
Jul 28 13:35:58.344336 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.344302    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1"} err="failed to get container status \"9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1\": rpc error: code = NotFound desc = could not find container \"9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1\": container with ID starting with 9b22c3f916034fe51fbb4e00ee793a47544f91f7643b348c850471d9870c7ab1 not found: ID does not exist"
Jul 28 13:35:58.344336 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.344320    2553 scope.go:117] "RemoveContainer" containerID="a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624"
Jul 28 13:35:58.344528 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:35:58.344514    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624\": container with ID starting with a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624 not found: ID does not exist" containerID="a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624"
Jul 28 13:35:58.344566 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.344532    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624"} err="failed to get container status \"a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624\": rpc error: code = NotFound desc = could not find container \"a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624\": container with ID starting with a61a579168329cacbfc86270bfa02a23b962f33e6a66b3a52ee797c658757624 not found: ID does not exist"
Jul 28 13:35:58.396470 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.396396    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8txzz\" (UniqueName: \"kubernetes.io/projected/82c22232-ba57-477d-812f-81bb8879c5dc-kube-api-access-8txzz\") pod \"82c22232-ba57-477d-812f-81bb8879c5dc\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") "
Jul 28 13:35:58.396470 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.396448    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82c22232-ba57-477d-812f-81bb8879c5dc-proxy-tls\") pod \"82c22232-ba57-477d-812f-81bb8879c5dc\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") "
Jul 28 13:35:58.396632 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.396493    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82c22232-ba57-477d-812f-81bb8879c5dc-kube-rbac-proxy-sar-config\") pod \"82c22232-ba57-477d-812f-81bb8879c5dc\" (UID: \"82c22232-ba57-477d-812f-81bb8879c5dc\") "
Jul 28 13:35:58.396962 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.396935    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c22232-ba57-477d-812f-81bb8879c5dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82c22232-ba57-477d-812f-81bb8879c5dc" (UID: "82c22232-ba57-477d-812f-81bb8879c5dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:35:58.398515 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.398489    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c22232-ba57-477d-812f-81bb8879c5dc-kube-api-access-8txzz" (OuterVolumeSpecName: "kube-api-access-8txzz") pod "82c22232-ba57-477d-812f-81bb8879c5dc" (UID: "82c22232-ba57-477d-812f-81bb8879c5dc"). InnerVolumeSpecName "kube-api-access-8txzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:35:58.398604 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.398512    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c22232-ba57-477d-812f-81bb8879c5dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82c22232-ba57-477d-812f-81bb8879c5dc" (UID: "82c22232-ba57-477d-812f-81bb8879c5dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:35:58.497844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.497807    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82c22232-ba57-477d-812f-81bb8879c5dc-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:35:58.497844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.497839    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82c22232-ba57-477d-812f-81bb8879c5dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:35:58.498016 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.497854    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8txzz\" (UniqueName: \"kubernetes.io/projected/82c22232-ba57-477d-812f-81bb8879c5dc-kube-api-access-8txzz\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:35:58.646735 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.646641    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"]
Jul 28 13:35:58.651044 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:35:58.651016    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b971b-predictor-75f98f6896-m2rmj"]
Jul 28 13:36:00.476021 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:36:00.475986    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" path="/var/lib/kubelet/pods/82c22232-ba57-477d-812f-81bb8879c5dc/volumes"
Jul 28 13:41:07.553526 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:41:07.553491    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:25.034310 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.034226    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:25.151512 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.151471    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:25.189192 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.189165    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"]
Jul 28 13:43:25.189408 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.189383    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container" containerID="cri-o://daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e" gracePeriod=30
Jul 28 13:43:25.189478 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.189422    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kube-rbac-proxy" containerID="cri-o://5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d" gracePeriod=30
Jul 28 13:43:25.229683 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.229652    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:25.840468 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840436    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m7876/must-gather-8mxp4"]
Jul 28 13:43:25.840780 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840767    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container"
Jul 28 13:43:25.840844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840781    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container"
Jul 28 13:43:25.840844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840804    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kube-rbac-proxy"
Jul 28 13:43:25.840844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840810    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kube-rbac-proxy"
Jul 28 13:43:25.840844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840825    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kube-rbac-proxy"
Jul 28 13:43:25.840844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840831    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kube-rbac-proxy"
Jul 28 13:43:25.840844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840840    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container"
Jul 28 13:43:25.840844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840845    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container"
Jul 28 13:43:25.841053 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840903    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kserve-container"
Jul 28 13:43:25.841053 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840911    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dcd5827-68af-4199-93d3-1729343138ff" containerName="kube-rbac-proxy"
Jul 28 13:43:25.841053 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840917    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kube-rbac-proxy"
Jul 28 13:43:25.841053 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.840927    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c22232-ba57-477d-812f-81bb8879c5dc" containerName="kserve-container"
Jul 28 13:43:25.844112 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.844089    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:43:25.844338 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.844279    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:25.847403 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.847383    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m7876\"/\"openshift-service-ca.crt\""
Jul 28 13:43:25.847483 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.847418    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m7876\"/\"default-dockercfg-z8bnr\""
Jul 28 13:43:25.848554 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.848529    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m7876\"/\"kube-root-ca.crt\""
Jul 28 13:43:25.856098 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.856043    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7876/must-gather-8mxp4"]
Jul 28 13:43:25.936275 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.936248    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:25.953340 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.953315    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ebc1ec0-4246-49da-8c9e-01f78168f40b-must-gather-output\") pod \"must-gather-8mxp4\" (UID: \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\") " pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:43:25.953417 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:25.953366    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qf8t\" (UniqueName: \"kubernetes.io/projected/6ebc1ec0-4246-49da-8c9e-01f78168f40b-kube-api-access-9qf8t\") pod \"must-gather-8mxp4\" (UID: \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\") " pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:43:26.054717 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:26.054687    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ebc1ec0-4246-49da-8c9e-01f78168f40b-must-gather-output\") pod \"must-gather-8mxp4\" (UID: \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\") " pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:43:26.055100 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:26.054736    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qf8t\" (UniqueName: \"kubernetes.io/projected/6ebc1ec0-4246-49da-8c9e-01f78168f40b-kube-api-access-9qf8t\") pod \"must-gather-8mxp4\" (UID: \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\") " pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:43:26.055100 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:26.055027    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ebc1ec0-4246-49da-8c9e-01f78168f40b-must-gather-output\") pod \"must-gather-8mxp4\" (UID: \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\") " pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:43:26.064304 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:26.064274    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qf8t\" (UniqueName: \"kubernetes.io/projected/6ebc1ec0-4246-49da-8c9e-01f78168f40b-kube-api-access-9qf8t\") pod \"must-gather-8mxp4\" (UID: \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\") " pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:43:26.163760 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:26.163730    2553 generic.go:356] "Generic (PLEG): container finished" podID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerID="5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d" exitCode=2
Jul 28 13:43:26.163927 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:26.163806    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" event={"ID":"316b03dc-65ce-4d8d-8f1f-3d44a049a440","Type":"ContainerDied","Data":"5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d"}
Jul 28 13:43:26.165996 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:26.165983    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:43:26.294965 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:26.294938    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7876/must-gather-8mxp4"]
Jul 28 13:43:26.297065 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:43:26.297040    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ebc1ec0_4246_49da_8c9e_01f78168f40b.slice/crio-5b8df05d96ea402238471308c5f330ed001ab3b58850a79aa7bd5f52bbd971cc WatchSource:0}: Error finding container 5b8df05d96ea402238471308c5f330ed001ab3b58850a79aa7bd5f52bbd971cc: Status 404 returned error can't find the container with id 5b8df05d96ea402238471308c5f330ed001ab3b58850a79aa7bd5f52bbd971cc
Jul 28 13:43:26.299059 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:26.299041    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:43:27.173888 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:27.173852    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7876/must-gather-8mxp4" event={"ID":"6ebc1ec0-4246-49da-8c9e-01f78168f40b","Type":"ContainerStarted","Data":"5b8df05d96ea402238471308c5f330ed001ab3b58850a79aa7bd5f52bbd971cc"}
Jul 28 13:43:27.797929 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:27.797556    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 13:43:28.458217 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.458157    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:43:28.577435 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.577404    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316b03dc-65ce-4d8d-8f1f-3d44a049a440-proxy-tls\") pod \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") "
Jul 28 13:43:28.577627 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.577484    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87txm\" (UniqueName: \"kubernetes.io/projected/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-api-access-87txm\") pod \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") "
Jul 28 13:43:28.577627 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.577586    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-rbac-proxy-sar-config\") pod \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\" (UID: \"316b03dc-65ce-4d8d-8f1f-3d44a049a440\") "
Jul 28 13:43:28.578173 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.578127    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "316b03dc-65ce-4d8d-8f1f-3d44a049a440" (UID: "316b03dc-65ce-4d8d-8f1f-3d44a049a440"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:43:28.579977 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.579941    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-api-access-87txm" (OuterVolumeSpecName: "kube-api-access-87txm") pod "316b03dc-65ce-4d8d-8f1f-3d44a049a440" (UID: "316b03dc-65ce-4d8d-8f1f-3d44a049a440"). InnerVolumeSpecName "kube-api-access-87txm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:43:28.580100 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.579970    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316b03dc-65ce-4d8d-8f1f-3d44a049a440-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "316b03dc-65ce-4d8d-8f1f-3d44a049a440" (UID: "316b03dc-65ce-4d8d-8f1f-3d44a049a440"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:43:28.678557 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.678527    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316b03dc-65ce-4d8d-8f1f-3d44a049a440-proxy-tls\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:43:28.678557 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.678553    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87txm\" (UniqueName: \"kubernetes.io/projected/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-api-access-87txm\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:43:28.678557 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:28.678565    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316b03dc-65ce-4d8d-8f1f-3d44a049a440-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:43:29.185936 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:29.185894    2553 generic.go:356] "Generic (PLEG): container finished" podID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerID="daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e" exitCode=0
Jul 28 13:43:29.186138 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:29.185953    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"
Jul 28 13:43:29.186138 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:29.185977    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" event={"ID":"316b03dc-65ce-4d8d-8f1f-3d44a049a440","Type":"ContainerDied","Data":"daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e"}
Jul 28 13:43:29.186138 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:29.186040    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp" event={"ID":"316b03dc-65ce-4d8d-8f1f-3d44a049a440","Type":"ContainerDied","Data":"8bfde4628ee0a04e50e974ff2fcbb6597356459808056a12a3375c39045d2933"}
Jul 28 13:43:29.186138 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:29.186079    2553 scope.go:117] "RemoveContainer" containerID="5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d"
Jul 28 13:43:29.211765 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:29.211736    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"]
Jul 28 13:43:29.215668 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:29.215642    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-18d3f-predictor-79b6b9f98b-fc2xp"]
Jul 28 13:43:30.477560 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:30.477526    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" path="/var/lib/kubelet/pods/316b03dc-65ce-4d8d-8f1f-3d44a049a440/volumes"
Jul 28 13:43:32.130365 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:32.130152    2553 scope.go:117] "RemoveContainer" containerID="daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e"
Jul 28 13:43:32.345027 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:32.345004    2553 scope.go:117] "RemoveContainer" containerID="5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d"
Jul 28 13:43:32.345353 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:43:32.345331    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d\": container with ID starting with 5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d not found: ID does not exist" containerID="5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d"
Jul 28 13:43:32.345449 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:32.345362    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d"} err="failed to get container status \"5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d\": rpc error: code = NotFound desc = could not find container \"5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d\": container with ID starting with 5c40dc4abc0ac6aaa02a7598204b5d4213db703dd0efaf174c2e3b2fbd538e7d not found: ID does not exist"
Jul 28 13:43:32.345449 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:32.345384    2553 scope.go:117] "RemoveContainer" containerID="daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e"
Jul 28 13:43:32.345659 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:43:32.345643    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e\": container with ID starting with daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e not found: ID does not exist" containerID="daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e"
Jul 28 13:43:32.345703 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:32.345662    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e"} err="failed to get container status \"daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e\": rpc error: code = NotFound desc = could not find container \"daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e\": container with ID starting with daf0504c4754c33603053b538ef7853c4425eefa8059fa41f80e8857609bea9e not found: ID does not exist"
Jul 28 13:43:33.208433 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:33.208399    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7876/must-gather-8mxp4" event={"ID":"6ebc1ec0-4246-49da-8c9e-01f78168f40b","Type":"ContainerStarted","Data":"b4bcfd4eb8ad1963a0c470fcbca1a55589dcfb63a4a6e6845f423946914b8d7d"}
Jul 28 13:43:33.208929 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:33.208446    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7876/must-gather-8mxp4" event={"ID":"6ebc1ec0-4246-49da-8c9e-01f78168f40b","Type":"ContainerStarted","Data":"494b5e521150a21df0196c691b3d641acaa1b27f22865893245c18afdd5fd0fe"}
Jul 28 13:43:33.229244 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:33.229186    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m7876/must-gather-8mxp4" podStartSLOduration=2.117314751 podStartE2EDuration="8.229172577s" totalImagesPullingTime="6.111857826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:43:33.22669998 +0000 UTC m=+2983.264068888" watchObservedRunningTime="2026-07-28 13:43:33.229172577 +0000 UTC m=+2983.266541497"
Jul 28 13:43:41.202533 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:41.202470    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:52.368364 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:52.368325    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:55.311389 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:55.311353    2553 generic.go:356] "Generic (PLEG): container finished" podID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" containerID="494b5e521150a21df0196c691b3d641acaa1b27f22865893245c18afdd5fd0fe" exitCode=0
Jul 28 13:43:55.311887 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:55.311425    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7876/must-gather-8mxp4" event={"ID":"6ebc1ec0-4246-49da-8c9e-01f78168f40b","Type":"ContainerDied","Data":"494b5e521150a21df0196c691b3d641acaa1b27f22865893245c18afdd5fd0fe"}
Jul 28 13:43:55.311887 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:55.311733    2553 scope.go:117] "RemoveContainer" containerID="494b5e521150a21df0196c691b3d641acaa1b27f22865893245c18afdd5fd0fe"
Jul 28 13:43:55.702857 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:55.702836    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m7876_must-gather-8mxp4_6ebc1ec0-4246-49da-8c9e-01f78168f40b/gather/0.log"
Jul 28 13:43:56.168921 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:56.168886    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:56.358093 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:56.358065    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:56.422160 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:56.422063    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:43:58.941317 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:58.941286    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2f4qz_2058fd10-4ec4-45cd-b41f-3496159fa93c/global-pull-secret-syncer/0.log"
Jul 28 13:43:59.081425 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:59.081393    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log"
Jul 28 13:43:59.214734 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:43:59.214655    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-149.ec2.internal_f5159b379a705575dacfde81933eecbe/haproxy/0.log"
Jul 28 13:44:01.199121 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.199093    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-m7876/must-gather-8mxp4"]
Jul 28 13:44:01.199592 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.199265    2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-m7876/must-gather-8mxp4" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" containerName="copy" containerID="cri-o://b4bcfd4eb8ad1963a0c470fcbca1a55589dcfb63a4a6e6845f423946914b8d7d" gracePeriod=2
Jul 28 13:44:01.204105 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.204079    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-m7876/must-gather-8mxp4"]
Jul 28 13:44:01.338035 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.338010    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m7876_must-gather-8mxp4_6ebc1ec0-4246-49da-8c9e-01f78168f40b/copy/0.log"
Jul 28 13:44:01.338425 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.338402    2553 generic.go:356] "Generic (PLEG): container finished" podID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" containerID="b4bcfd4eb8ad1963a0c470fcbca1a55589dcfb63a4a6e6845f423946914b8d7d" exitCode=143
Jul 28 13:44:01.429893 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.429866    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m7876_must-gather-8mxp4_6ebc1ec0-4246-49da-8c9e-01f78168f40b/copy/0.log"
Jul 28 13:44:01.431317 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.431302    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:44:01.433593 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:44:01.433571    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8mxp4\" is forbidden: User \"system:node:ip-10-0-135-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m7876\": no relationship found between node 'ip-10-0-135-149.ec2.internal' and this object" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:44:01.566262 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.566167    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qf8t\" (UniqueName: \"kubernetes.io/projected/6ebc1ec0-4246-49da-8c9e-01f78168f40b-kube-api-access-9qf8t\") pod \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\" (UID: \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\") "
Jul 28 13:44:01.566419 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.566267    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ebc1ec0-4246-49da-8c9e-01f78168f40b-must-gather-output\") pod \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\" (UID: \"6ebc1ec0-4246-49da-8c9e-01f78168f40b\") "
Jul 28 13:44:01.567833 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.567785    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebc1ec0-4246-49da-8c9e-01f78168f40b-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "6ebc1ec0-4246-49da-8c9e-01f78168f40b" (UID: "6ebc1ec0-4246-49da-8c9e-01f78168f40b"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:44:01.568425 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.568399    2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebc1ec0-4246-49da-8c9e-01f78168f40b-kube-api-access-9qf8t" (OuterVolumeSpecName: "kube-api-access-9qf8t") pod "6ebc1ec0-4246-49da-8c9e-01f78168f40b" (UID: "6ebc1ec0-4246-49da-8c9e-01f78168f40b"). InnerVolumeSpecName "kube-api-access-9qf8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:44:01.667755 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.667722    2553 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ebc1ec0-4246-49da-8c9e-01f78168f40b-must-gather-output\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:44:01.667755 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.667748    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qf8t\" (UniqueName: \"kubernetes.io/projected/6ebc1ec0-4246-49da-8c9e-01f78168f40b-kube-api-access-9qf8t\") on node \"ip-10-0-135-149.ec2.internal\" DevicePath \"\""
Jul 28 13:44:01.980879 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:01.980848    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:44:02.068044 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:02.068010    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n6sj9_ba0380da-f7c1-46d2-beca-b5475dcc58bb/konnectivity-agent/0.log" limit=1024
Jul 28 13:44:02.343329 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:02.343245    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m7876_must-gather-8mxp4_6ebc1ec0-4246-49da-8c9e-01f78168f40b/copy/0.log"
Jul 28 13:44:02.343728 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:02.343641    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:44:02.343728 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:02.343676    2553 scope.go:117] "RemoveContainer" containerID="b4bcfd4eb8ad1963a0c470fcbca1a55589dcfb63a4a6e6845f423946914b8d7d"
Jul 28 13:44:02.346742 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:44:02.346712    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8mxp4\" is forbidden: User \"system:node:ip-10-0-135-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m7876\": no relationship found between node 'ip-10-0-135-149.ec2.internal' and this object" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:44:02.351483 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:02.351463    2553 scope.go:117] "RemoveContainer" containerID="494b5e521150a21df0196c691b3d641acaa1b27f22865893245c18afdd5fd0fe"
Jul 28 13:44:02.353924 ip-10-0-135-149 kubenswrapper[2553]: E0728 13:44:02.353897    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8mxp4\" is forbidden: User \"system:node:ip-10-0-135-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m7876\": no relationship found between node 'ip-10-0-135-149.ec2.internal' and this object" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" pod="openshift-must-gather-m7876/must-gather-8mxp4"
Jul 28 13:44:02.475738 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:02.475715    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" path="/var/lib/kubelet/pods/6ebc1ec0-4246-49da-8c9e-01f78168f40b/volumes"
Jul 28 13:44:03.482444 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.482413    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f15b3ac1-60d8-4b82-b686-9e52b6d23868/alertmanager/0.log"
Jul 28 13:44:03.504500 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.504474    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f15b3ac1-60d8-4b82-b686-9e52b6d23868/config-reloader/0.log"
Jul 28 13:44:03.527127 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.527104    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f15b3ac1-60d8-4b82-b686-9e52b6d23868/kube-rbac-proxy-web/0.log"
Jul 28 13:44:03.550907 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.550887    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f15b3ac1-60d8-4b82-b686-9e52b6d23868/kube-rbac-proxy/0.log"
Jul 28 13:44:03.573677 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.573651    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f15b3ac1-60d8-4b82-b686-9e52b6d23868/kube-rbac-proxy-metric/0.log"
Jul 28 13:44:03.596371 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.596350    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f15b3ac1-60d8-4b82-b686-9e52b6d23868/prom-label-proxy/0.log"
Jul 28 13:44:03.621851 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.621826    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f15b3ac1-60d8-4b82-b686-9e52b6d23868/init-config-reloader/0.log"
Jul 28 13:44:03.705571 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.705546    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-kp9k5_6a73e5a1-f9e8-4c93-9790-b1315e823e13/kube-state-metrics/0.log"
Jul 28 13:44:03.726690 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.726655    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-kp9k5_6a73e5a1-f9e8-4c93-9790-b1315e823e13/kube-rbac-proxy-main/0.log"
Jul 28 13:44:03.747458 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.747434    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-kp9k5_6a73e5a1-f9e8-4c93-9790-b1315e823e13/kube-rbac-proxy-self/0.log"
Jul 28 13:44:03.772936 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.772918    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bf495b5d4-v98jh_6afed194-561e-454b-a632-187cd39d4a22/metrics-server/0.log"
Jul 28 13:44:03.800915 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.800891    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-g9mcw_aac227a6-52b5-41f6-8cce-f83aac850e25/monitoring-plugin/0.log"
Jul 28 13:44:03.830019 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.830001    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6drlp_addd1058-8188-4cb5-b23a-e25a8dfb393c/node-exporter/0.log"
Jul 28 13:44:03.852566 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.852536    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6drlp_addd1058-8188-4cb5-b23a-e25a8dfb393c/kube-rbac-proxy/0.log"
Jul 28 13:44:03.881177 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:03.881160    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6drlp_addd1058-8188-4cb5-b23a-e25a8dfb393c/init-textfile/0.log"
Jul 28 13:44:04.090210 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.090132    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-srvzp_a710ec85-fda1-47b0-a06c-6d5539f709c8/kube-rbac-proxy-main/0.log"
Jul 28 13:44:04.112249 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.112222    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-srvzp_a710ec85-fda1-47b0-a06c-6d5539f709c8/kube-rbac-proxy-self/0.log"
Jul 28 13:44:04.133750 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.133725    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-srvzp_a710ec85-fda1-47b0-a06c-6d5539f709c8/openshift-state-metrics/0.log"
Jul 28 13:44:04.173244 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.173219    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe2553f-bf89-4968-9a57-aa3629efa2d2/prometheus/0.log"
Jul 28 13:44:04.189501 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.189481    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe2553f-bf89-4968-9a57-aa3629efa2d2/config-reloader/0.log"
Jul 28 13:44:04.211731 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.211714    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe2553f-bf89-4968-9a57-aa3629efa2d2/thanos-sidecar/0.log"
Jul 28 13:44:04.232623 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.232606    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe2553f-bf89-4968-9a57-aa3629efa2d2/kube-rbac-proxy-web/0.log"
Jul 28 13:44:04.257605 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.257581    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe2553f-bf89-4968-9a57-aa3629efa2d2/kube-rbac-proxy/0.log"
Jul 28 13:44:04.276842 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.276824    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe2553f-bf89-4968-9a57-aa3629efa2d2/kube-rbac-proxy-thanos/0.log"
Jul 28 13:44:04.297358 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.297337    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe2553f-bf89-4968-9a57-aa3629efa2d2/init-config-reloader/0.log"
Jul 28 13:44:04.327635 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.327613    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-btb6q_b3b3e3df-3308-4e68-b285-d82cbce5fd3e/prometheus-operator/0.log"
Jul 28 13:44:04.342421 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.342356    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-btb6q_b3b3e3df-3308-4e68-b285-d82cbce5fd3e/kube-rbac-proxy/0.log"
Jul 28 13:44:04.406252 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.406225    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67c8bd987c-rp7hj_db890d24-1a50-4d69-8909-c7f2d4cb89ef/telemeter-client/0.log"
Jul 28 13:44:04.426175 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.426151    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67c8bd987c-rp7hj_db890d24-1a50-4d69-8909-c7f2d4cb89ef/reload/0.log"
Jul 28 13:44:04.449030 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.449008    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67c8bd987c-rp7hj_db890d24-1a50-4d69-8909-c7f2d4cb89ef/kube-rbac-proxy/0.log"
Jul 28 13:44:04.481732 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.481706    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8fdfcf97-7h2kz_19fad7fd-e532-475d-b891-db67a414256c/thanos-query/0.log"
Jul 28 13:44:04.503968 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.503940    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8fdfcf97-7h2kz_19fad7fd-e532-475d-b891-db67a414256c/kube-rbac-proxy-web/0.log"
Jul 28 13:44:04.523487 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.523468    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8fdfcf97-7h2kz_19fad7fd-e532-475d-b891-db67a414256c/kube-rbac-proxy/0.log"
Jul 28 13:44:04.543613 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.543586    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8fdfcf97-7h2kz_19fad7fd-e532-475d-b891-db67a414256c/prom-label-proxy/0.log"
Jul 28 13:44:04.566549 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.566524    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8fdfcf97-7h2kz_19fad7fd-e532-475d-b891-db67a414256c/kube-rbac-proxy-rules/0.log"
Jul 28 13:44:04.586500 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:04.586482    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8fdfcf97-7h2kz_19fad7fd-e532-475d-b891-db67a414256c/kube-rbac-proxy-metrics/0.log"
Jul 28 13:44:06.862380 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:06.862348    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-cn6x5_111c52cc-e449-4068-b7ed-a1af6ad8b5f1/download-server/0.log"
Jul 28 13:44:07.274687 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.274651    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"]
Jul 28 13:44:07.275039 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275026    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" containerName="gather"
Jul 28 13:44:07.275082 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275041    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" containerName="gather"
Jul 28 13:44:07.275082 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275057    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container"
Jul 28 13:44:07.275082 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275063    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container"
Jul 28 13:44:07.275082 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275078    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kube-rbac-proxy"
Jul 28 13:44:07.275208 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275084    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kube-rbac-proxy"
Jul 28 13:44:07.275208 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275092    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" containerName="copy"
Jul 28 13:44:07.275208 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275096    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" containerName="copy"
Jul 28 13:44:07.275208 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275146    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" containerName="gather"
Jul 28 13:44:07.275208 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275156    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebc1ec0-4246-49da-8c9e-01f78168f40b" containerName="copy"
Jul 28 13:44:07.275208 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275162    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kube-rbac-proxy"
Jul 28 13:44:07.275208 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.275168    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="316b03dc-65ce-4d8d-8f1f-3d44a049a440" containerName="kserve-container"
Jul 28 13:44:07.278926 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.278905    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.281763 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.281744    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95qqx\"/\"openshift-service-ca.crt\""
Jul 28 13:44:07.281878 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.281745    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-95qqx\"/\"default-dockercfg-7wr6f\""
Jul 28 13:44:07.282891 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.282874    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95qqx\"/\"kube-root-ca.crt\""
Jul 28 13:44:07.287951 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.287923    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"]
Jul 28 13:44:07.415629 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.415590    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7hj\" (UniqueName: \"kubernetes.io/projected/9c9396a5-1a0a-4825-9672-ea16ee817350-kube-api-access-qc7hj\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.415629 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.415631    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-sys\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.415880 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.415653    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-proc\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.415880 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.415751    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-lib-modules\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.415880 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.415778    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-podres\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.516623 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.516592    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-lib-modules\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.516623 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.516623    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-podres\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.516866 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.516669    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7hj\" (UniqueName: \"kubernetes.io/projected/9c9396a5-1a0a-4825-9672-ea16ee817350-kube-api-access-qc7hj\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.516866 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.516685    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-sys\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.516866 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.516698    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-proc\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.516866 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.516755    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-lib-modules\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.516866 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.516759    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-proc\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.516866 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.516836    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-sys\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.516866 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.516856    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c9396a5-1a0a-4825-9672-ea16ee817350-podres\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.525026 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.524978    2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7hj\" (UniqueName: \"kubernetes.io/projected/9c9396a5-1a0a-4825-9672-ea16ee817350-kube-api-access-qc7hj\") pod \"perf-node-gather-daemonset-s64rs\" (UID: \"9c9396a5-1a0a-4825-9672-ea16ee817350\") " pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.589696 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.589637    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:07.725844 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:07.725815    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"]
Jul 28 13:44:07.727820 ip-10-0-135-149 kubenswrapper[2553]: W0728 13:44:07.727778    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9c9396a5_1a0a_4825_9672_ea16ee817350.slice/crio-ff0409aff1ed860a7548c5d7f48155f652cb3fe77145ed3025d00c459f21c611 WatchSource:0}: Error finding container ff0409aff1ed860a7548c5d7f48155f652cb3fe77145ed3025d00c459f21c611: Status 404 returned error can't find the container with id ff0409aff1ed860a7548c5d7f48155f652cb3fe77145ed3025d00c459f21c611
Jul 28 13:44:08.248664 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:08.248632    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hlrq_1725782f-575b-4eea-bca2-58b8de98444d/dns/0.log"
Jul 28 13:44:08.268265 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:08.268236    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hlrq_1725782f-575b-4eea-bca2-58b8de98444d/kube-rbac-proxy/0.log"
Jul 28 13:44:08.370201 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:08.370162    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs" event={"ID":"9c9396a5-1a0a-4825-9672-ea16ee817350","Type":"ContainerStarted","Data":"e52f01c80df29f6371a2d673921a76da89b5651fde35bce69625830c264bd0e4"}
Jul 28 13:44:08.370407 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:08.370217    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs" event={"ID":"9c9396a5-1a0a-4825-9672-ea16ee817350","Type":"ContainerStarted","Data":"ff0409aff1ed860a7548c5d7f48155f652cb3fe77145ed3025d00c459f21c611"}
Jul 28 13:44:08.370407 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:08.370248    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:08.386663 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:08.386619    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs" podStartSLOduration=1.386603859 podStartE2EDuration="1.386603859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:08.385890544 +0000 UTC m=+3018.423259457" watchObservedRunningTime="2026-07-28 13:44:08.386603859 +0000 UTC m=+3018.423972807"
Jul 28 13:44:08.430010 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:08.429986    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w497j_302c2c0a-0dd0-4af7-83fb-f52dd09da666/dns-node-resolver/0.log"
Jul 28 13:44:08.952852 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:08.952824    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzt5m_c2439221-c604-428e-885e-84788d713251/node-ca/0.log"
Jul 28 13:44:10.016221 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:10.016185    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mzdrz_111434a7-0841-42b1-85b1-cd16ab0eada9/serve-healthcheck-canary/0.log"
Jul 28 13:44:10.522419 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:10.522386    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-td7h2_59199dc1-6c80-47c3-a6a4-fb9bcef71268/kube-rbac-proxy/0.log"
Jul 28 13:44:10.540280 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:10.540255    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-td7h2_59199dc1-6c80-47c3-a6a4-fb9bcef71268/exporter/0.log"
Jul 28 13:44:10.560009 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:10.559987    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-td7h2_59199dc1-6c80-47c3-a6a4-fb9bcef71268/extractor/0.log"
Jul 28 13:44:13.031817 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:13.031769    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-fx5b6_c2fcbdd3-610b-4bc7-8c11-e49961937aa0/s3-init/0.log"
Jul 28 13:44:13.046354 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:13.046330    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-fx5b6_c2fcbdd3-610b-4bc7-8c11-e49961937aa0/download-hf-model/0.log"
Jul 28 13:44:13.075916 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:13.075890    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-lpddd_25cea127-7ab1-41f8-a26b-e90fcf628800/seaweedfs/0.log"
Jul 28 13:44:14.384445 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:14.384410    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-95qqx/perf-node-gather-daemonset-s64rs"
Jul 28 13:44:18.476133 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:18.476070    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5wdsp_14d239a1-4dfe-4102-b8e5-ddc411b60264/kube-multus-additional-cni-plugins/0.log"
Jul 28 13:44:18.497785 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:18.497762    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5wdsp_14d239a1-4dfe-4102-b8e5-ddc411b60264/egress-router-binary-copy/0.log"
Jul 28 13:44:18.517539 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:18.517516    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5wdsp_14d239a1-4dfe-4102-b8e5-ddc411b60264/cni-plugins/0.log"
Jul 28 13:44:18.536636 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:18.536614    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5wdsp_14d239a1-4dfe-4102-b8e5-ddc411b60264/bond-cni-plugin/0.log"
Jul 28 13:44:18.555517 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:18.555498    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5wdsp_14d239a1-4dfe-4102-b8e5-ddc411b60264/routeoverride-cni/0.log"
Jul 28 13:44:18.574731 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:18.574709    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5wdsp_14d239a1-4dfe-4102-b8e5-ddc411b60264/whereabouts-cni-bincopy/0.log"
Jul 28 13:44:18.593804 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:18.593772    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5wdsp_14d239a1-4dfe-4102-b8e5-ddc411b60264/whereabouts-cni/0.log"
Jul 28 13:44:19.016453 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:19.016428    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jf286_812e57fd-70cf-4e3d-be5a-b2ad7fca3c21/kube-multus/0.log"
Jul 28 13:44:19.083632 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:19.083595    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qrq22_1bece3c2-4eb4-4219-a757-45a57c351842/network-metrics-daemon/0.log"
Jul 28 13:44:19.101317 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:19.101296    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qrq22_1bece3c2-4eb4-4219-a757-45a57c351842/kube-rbac-proxy/0.log"
Jul 28 13:44:20.021255 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:20.021225    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hmhh_51477dbb-d4b7-448e-9583-64f2d909364d/ovn-controller/0.log"
Jul 28 13:44:20.048917 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:20.048879    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hmhh_51477dbb-d4b7-448e-9583-64f2d909364d/ovn-acl-logging/0.log"
Jul 28 13:44:20.077636 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:20.077614    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hmhh_51477dbb-d4b7-448e-9583-64f2d909364d/kube-rbac-proxy-node/0.log"
Jul 28 13:44:20.101810 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:20.101770    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hmhh_51477dbb-d4b7-448e-9583-64f2d909364d/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 13:44:20.120308 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:20.120287    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hmhh_51477dbb-d4b7-448e-9583-64f2d909364d/northd/0.log"
Jul 28 13:44:20.140548 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:20.140525    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hmhh_51477dbb-d4b7-448e-9583-64f2d909364d/nbdb/0.log"
Jul 28 13:44:20.162833 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:20.162814    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hmhh_51477dbb-d4b7-448e-9583-64f2d909364d/sbdb/0.log"
Jul 28 13:44:20.226053 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:20.226027    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hmhh_51477dbb-d4b7-448e-9583-64f2d909364d/ovnkube-controller/0.log"
Jul 28 13:44:21.691223 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:21.691191    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-cqxxn_2ebe21a0-6999-409b-a63b-0fd0265d25b5/network-check-target-container/0.log"
Jul 28 13:44:22.596976 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:22.596946    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-tkkwh_14061f83-1fc2-4769-8f1a-ebd040d3e4e6/iptables-alerter/0.log"
Jul 28 13:44:23.242252 ip-10-0-135-149 kubenswrapper[2553]: I0728 13:44:23.242224    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-pr9jm_89cc2b6b-5922-4400-8607-99a5ff7485f0/tuned/0.log"