Sep 08 13:16:27.189676 ip-10-0-140-239 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 13:16:27.565742 ip-10-0-140-239 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:27.565742 ip-10-0-140-239 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 13:16:27.565742 ip-10-0-140-239 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:27.565742 ip-10-0-140-239 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 13:16:27.565742 ip-10-0-140-239 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:27.567958 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.567871    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 13:16:27.573770 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573746    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:27.573770 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573765    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:27.573770 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573769    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:27.573770 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573772    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:27.573770 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573776    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:27.573770 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573778    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573782    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573785    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573787    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573790    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573793    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573796    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573799    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573801    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573804    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573806    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573809    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573812    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573814    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573817    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573820    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573823    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573826    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573828    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573831    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:27.573998 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573833    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573839    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573842    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573845    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573848    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573850    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573852    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573855    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573858    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573860    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573863    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573865    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573868    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573871    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573874    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573878    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573882    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573885    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573888    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:27.574497 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573891    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573894    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573896    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573900    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573904    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573906    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573909    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573911    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573914    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573916    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573919    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573922    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573924    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573927    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573930    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573933    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573936    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573938    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573941    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:27.574959 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573944    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573946    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573948    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573952    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573956    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573958    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573961    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573964    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573967    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573970    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573973    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573976    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573978    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573981    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573983    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573986    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573988    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573991    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573994    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.573997    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:27.575439 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574002    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574005    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574008    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574011    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574013    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574016    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574019    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574022    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574024    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574028    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574031    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574035    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574038    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574040    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574043    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574047    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574774    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574782    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574785    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:27.575991 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574788    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574791    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574794    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574797    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574799    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574802    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574805    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574807    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574810    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574812    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574817    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574820    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574823    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574826    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574829    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574833    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574836    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574839    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574841    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:27.576480 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574844    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574847    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574849    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574852    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574854    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574857    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574860    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574862    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574865    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574868    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574871    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574874    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574877    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574879    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574882    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574885    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574888    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574890    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574893    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574895    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:27.576945 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574898    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574900    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574903    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574905    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574908    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574911    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574913    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574916    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574919    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574921    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574924    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574926    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574929    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574931    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574934    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574936    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574939    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574941    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574944    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574947    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:27.577442 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574949    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574953    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574955    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574958    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574961    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574964    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574966    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574970    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574973    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574975    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574978    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574981    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574984    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574986    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574989    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574992    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574994    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574997    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.574999    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575002    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:27.577936 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575004    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575010    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575014    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575020    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575022    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575025    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575028    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575030    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575033    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575035    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575038    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575041    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575044    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575047    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575050    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575052    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575055    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575481    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575488    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:27.578435 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575492    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575495    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575498    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575501    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575504    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575506    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575509    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575512    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575514    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575517    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575520    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575523    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575526    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575528    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575532    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575538    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575541    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575543    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575547    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575551    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:27.578896 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575554    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575557    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575560    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575564    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575568    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575571    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575574    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575577    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575579    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575582    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575585    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575588    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575591    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575593    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575596    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575599    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575601    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575604    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575606    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:27.579380 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575609    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575612    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575614    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575617    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575619    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575622    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575624    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575626    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575631    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575634    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575637    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575639    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575642    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575645    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575647    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575650    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575652    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575655    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575658    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575661    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:27.579902 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575663    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575666    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575668    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575671    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575673    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575676    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575678    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575681    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575683    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575685    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575690    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575693    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575696    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575698    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575701    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575703    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575706    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575708    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575711    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575713    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:27.580424 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575717    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575720    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575723    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575727    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575729    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575732    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575734    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575737    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575739    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575742    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575745    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575747    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575749    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575752    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575754    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575757    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575759    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.575762    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575835    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575876    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575884    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 13:16:27.580958 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575888    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575892    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575896    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575900    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575904    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575907    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575910    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575913    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575917    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575920    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575923    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575927    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575930    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575934    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575938    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575942    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575945    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575948    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575951    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575954    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575958    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575960    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575963    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 13:16:27.581484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575966    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575970    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575973    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575991    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575995    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.575999    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576003    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576007    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576010    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576012    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576015    2580 flags.go:64] FLAG: --enable-server="true"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576018    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576023    2580 flags.go:64] FLAG: --event-burst="100"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576026    2580 flags.go:64] FLAG: --event-qps="50"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576029    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576032    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576035    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576039    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576042    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576045    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576048    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576051    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576054    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576062    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576065    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 13:16:27.582042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576068    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576071    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576074    2580 flags.go:64] FLAG: --feature-gates=""
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576077    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576080    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576083    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576086    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576090    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576092    2580 flags.go:64] FLAG: --help="false"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576095    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576098    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576101    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576104    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576107    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576111    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576113    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576116    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576119    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576122    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576125    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576128    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576131    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576134    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576137    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 13:16:27.582724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576140    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576142    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576145    2580 flags.go:64] FLAG: --lock-file=""
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576148    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576151    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576154    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576161    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576164    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576167    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576170    2580 flags.go:64] FLAG: --logging-format="text"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576173    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576176    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576179    2580 flags.go:64] FLAG: --manifest-url=""
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576182    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576187    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576190    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576194    2580 flags.go:64] FLAG: --max-pods="110"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576197    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576200    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576203    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576206    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576209    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576212    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576214    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576222    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 13:16:27.583324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576225    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576228    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576231    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576233    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576239    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576242    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576245    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576248    2580 flags.go:64] FLAG: --port="10250"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576250    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576254    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c5f07b632abea0a3"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576257    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576260    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576263    2580 flags.go:64] FLAG: --register-node="true"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576267    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576271    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576274    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576277    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576280    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576284    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576287    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576290    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576293    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576296    2580 flags.go:64] FLAG: --runonce="false"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576299    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576302    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 13:16:27.583952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576304    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576308    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576310    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576313    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576316    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576319    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576322    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576325    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576328    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576331    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576334    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576336    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576339    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576345    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576348    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576351    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576355    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576358    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576361    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576363    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576367    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576370    2580 flags.go:64] FLAG: --v="2"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576374    2580 flags.go:64] FLAG: --version="false"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576379    2580 flags.go:64] FLAG: --vmodule=""
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576383    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 13:16:27.584620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.576386    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576523    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576528    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576532    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576535    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576538    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576541    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576544    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576547    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576550    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576552    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576555    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576559    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576563    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576566    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576569    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576572    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576575    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576577    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576580    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:27.585227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576584    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576587    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576589    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576592    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576594    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576597    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576600    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576604    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576607    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576610    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576612    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576617    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576621    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576624    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576627    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576629    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576632    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576634    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576637    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:27.585740 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576640    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576642    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576645    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576648    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576651    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576653    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576656    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576658    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576661    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576664    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576666    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576669    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576671    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576675    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576678    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576682    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576685    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576688    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576691    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576693    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:27.586206 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576697    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576700    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576702    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576705    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576707    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576711    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576713    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576716    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576719    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576721    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576724    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576727    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576729    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576732    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576734    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576737    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576739    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576742    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576744    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576747    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:27.586793 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576749    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576752    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576755    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576757    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576760    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576763    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576766    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576769    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576771    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576774    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576776    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576779    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576784    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576786    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576789    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576792    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576794    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576797    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576800    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576802    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:27.587292 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.576805    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.577555    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.584694    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.584712    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.585458    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.585470    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585531    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585536    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585540    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585543    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585546    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585549    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585552    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585555    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:27.587837 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585558    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585560    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585563    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585566    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585568    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585571    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585574    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585577    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585579    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585582    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585585    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585587    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585590    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585592    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585595    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585597    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585600    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585603    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585606    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585608    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:27.588198 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585612    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585620    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585623    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585626    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585629    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585632    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585635    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585637    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585640    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585643    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585646    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585648    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585651    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585653    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585656    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585659    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585661    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585664    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585667    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585669    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:27.588718 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585672    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585674    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585677    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585679    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585682    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585684    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585687    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585690    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585692    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585695    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585697    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585700    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585702    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585707    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585710    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585713    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585716    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585718    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585721    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585723    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:27.589219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585726    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585728    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585731    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585733    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585736    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585738    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585742    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585746    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585749    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585752    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585755    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585758    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585760    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585763    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585766    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585768    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585771    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585774    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585776    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:27.589750 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585779    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585782    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585786    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585789    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585792    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585795    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585799    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585802    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585805    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585808    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585810    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585813    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.585818    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585934    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:27.590227 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585939    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585943    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585945    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585948    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585952    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585955    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585957    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585960    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585963    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585965    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585968    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585970    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585973    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585976    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585978    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585981    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585983    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585985    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585988    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585990    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:27.590593 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585993    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585995    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.585998    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586000    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586003    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586006    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586008    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586012    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586014    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586017    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586020    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586022    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586025    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586027    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586030    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586032    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586035    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586039    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586043    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:27.591094 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586045    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586048    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586051    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586054    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586057    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586059    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586063    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586066    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586069    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586071    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586074    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586076    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586079    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586082    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586084    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586087    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586089    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586092    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586095    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586097    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:27.591575 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586100    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586103    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586106    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586110    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586113    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586116    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586118    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586121    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586123    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586126    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586128    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586131    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586133    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586136    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586138    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586141    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586143    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586146    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586148    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586151    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:27.592051 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586153    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586156    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586158    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586161    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586164    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586167    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586169    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586172    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586175    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586177    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586180    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586182    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586185    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586187    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586190    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586193    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586196    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586198    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:27.592551 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:27.586201    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:27.592987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.586206    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:27.592987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.587033    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 13:16:27.592987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.589052    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 13:16:27.592987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.589913    2580 server.go:1019] "Starting client certificate rotation"
Sep 08 13:16:27.592987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.590012    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:27.592987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.590041    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:27.592987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.592361    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:27.594906 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.594886    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:27.605347 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.605321    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 08 13:16:27.610567 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.610550    2580 log.go:25] "Validated CRI v1 image API"
Sep 08 13:16:27.611404 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.611370    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 13:16:27.615010 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.614983    2580 fs.go:135] Filesystem UUIDs: map[22f979fb-8a5f-4905-8f0c-4f1f6e2c18a2:/dev/nvme0n1p3 3d4fdf59-7749-4c68-85a2-b3afa331bb49:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 08 13:16:27.615083 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.615009    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 13:16:27.620971 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.620951    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:27.621162 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.621038    2580 manager.go:216] Machine: {Timestamp:2026-09-08 13:16:27.619830413 +0000 UTC m=+0.319761037 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116701 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec263e5f5375f4f9d03d6daa855a38f8 SystemUUID:ec263e5f-5375-f4f9-d03d-6daa855a38f8 BootID:e1a23ac8-f6a4-4477-b4a7-78346bcad7a7 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:f6:01:2e:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:f6:01:2e:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:5a:81:f0:0c:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 13:16:27.621162 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.621160    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 13:16:27.621269 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.621252    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 13:16:27.623860 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.623836    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 13:16:27.624040 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.623864    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-239.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 13:16:27.625138 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.625125    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 13:16:27.625205 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.625145    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 13:16:27.625205 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.625156    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:27.625989 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.625977    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:27.627194 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.627182    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 13:16:27.630225 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.630211    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:27.630371 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.630361    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 13:16:27.630455 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.630419    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 13:16:27.630455 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.630435    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 13:16:27.630455 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.630450    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 13:16:27.630574 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.630464    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 08 13:16:27.630574 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.630477    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 13:16:27.630870 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.630856    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:27.630987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.630878    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:27.632917 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.632900    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 13:16:27.635061 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.635049    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 13:16:27.635110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.635078    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 13:16:27.635951 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.635938    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 13:16:27.636008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.635955    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 13:16:27.636008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.635962    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 13:16:27.636008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.635967    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 13:16:27.636008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.635976    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 13:16:27.636008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.635982    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 13:16:27.636008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.635995    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 13:16:27.636008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.636000    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 13:16:27.636008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.636006    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 13:16:27.636008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.636012    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 13:16:27.636241 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.636020    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 13:16:27.636241 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.636026    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 13:16:27.636629 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.636614    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 13:16:27.636629 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.636625    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 13:16:27.640256 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.640232    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-239.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 13:16:27.640605 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.640585    2580 server.go:1268] "Started kubelet"
Sep 08 13:16:27.640683 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.640617    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 13:16:27.640907 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.640872    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 13:16:27.640992 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.640934    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 13:16:27.641088 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.640578    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 13:16:27.641499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.641472    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 13:16:27.641729 ip-10-0-140-239 systemd[1]: Started Kubernetes Kubelet.
Sep 08 13:16:27.642215 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.642145    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 13:16:27.646585 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.646431    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 13:16:27.646696 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.646679    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 13:16:27.647780 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.647764    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:27.648291 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.647414    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-239.ec2.internal.18d35ade4276fef1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-239.ec2.internal,UID:ip-10-0-140-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-239.ec2.internal,},FirstTimestamp:2026-09-08 13:16:27.640553201 +0000 UTC m=+0.340483826,LastTimestamp:2026-09-08 13:16:27.640553201 +0000 UTC m=+0.340483826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-239.ec2.internal,}"
Sep 08 13:16:27.648378 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.648347    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 13:16:27.648447 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.648433    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:27.649036 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649017    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 13:16:27.649036 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649019    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 13:16:27.649173 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649045    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 13:16:27.649173 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649140    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 13:16:27.649173 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649150    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 13:16:27.649295 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649224    2580 factory.go:55] Registering systemd factory
Sep 08 13:16:27.649295 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649237    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 08 13:16:27.649463 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649453    2580 factory.go:153] Registering CRI-O factory
Sep 08 13:16:27.649520 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649464    2580 factory.go:223] Registration of the crio container factory successfully
Sep 08 13:16:27.649520 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649515    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 13:16:27.649590 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649529    2580 factory.go:103] Registering Raw factory
Sep 08 13:16:27.649590 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.649539    2580 manager.go:1196] Started watching for new ooms in manager
Sep 08 13:16:27.650569 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.650555    2580 manager.go:349] Starting recovery of all containers
Sep 08 13:16:27.658436 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.658389    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nrwqz"
Sep 08 13:16:27.658928 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.658911    2580 manager.go:354] Recovery completed
Sep 08 13:16:27.659990 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.659960    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 13:16:27.660940 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.660003    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 13:16:27.663313 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.663301    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:27.664937 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.664923    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:27.665022 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.664956    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:27.665022 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.664970    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:27.665621 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.665606    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 13:16:27.665621 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.665620    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 13:16:27.665732 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.665638    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:27.665732 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.665670    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:27.665732 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.665694    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nrwqz"
Sep 08 13:16:27.666331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.666315    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:27.666386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.666341    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:27.666386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.666351    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:27.667007 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.666927    2580 policy_none.go:49] "None policy: Start"
Sep 08 13:16:27.667007 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.666945    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 13:16:27.667007 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.666959    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 13:16:27.670119 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.670105    2580 policy_none.go:47] "Start"
Sep 08 13:16:27.703051 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.703016    2580 manager.go:337] "Starting Device Plugin manager"
Sep 08 13:16:27.715358 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.703074    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 13:16:27.715358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.703088    2580 server.go:87] "Starting device plugin registration server"
Sep 08 13:16:27.715358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.703465    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 13:16:27.715358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.703478    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 13:16:27.715358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.703577    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 13:16:27.715358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.703663    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 13:16:27.715358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.703672    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 13:16:27.715358 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.704186    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 13:16:27.715358 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.704239    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:27.742998 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.742953    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 13:16:27.744269 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.744252    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 13:16:27.744354 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.744313    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 13:16:27.744354 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.744342    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 13:16:27.744443 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.744386    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 13:16:27.746465 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.746444    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:27.804314 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.804279    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:27.805711 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.805685    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:27.805836 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.805723    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:27.805836 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.805737    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:27.805836 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.805763    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.813349 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.813330    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.813454 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.813359    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-239.ec2.internal\": node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:27.829655 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.829568    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:27.845411 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.845374    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal"]
Sep 08 13:16:27.845542 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.845479    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:27.846455 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.846440    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:27.846532 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.846471    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:27.846532 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.846484    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:27.847929 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.847917    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:27.848041 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.848027    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.848082 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.848066    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:27.848693 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.848677    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:27.848754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.848704    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:27.848754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.848678    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:27.848754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.848719    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:27.848754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.848732    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:27.848754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.848746    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:27.850025 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.850010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.850069 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.850041    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:27.850185 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.850170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81546329125905c90a6831ec0139b393-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"81546329125905c90a6831ec0139b393\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.850225 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.850202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2bc47ecdba8650fc42750ab52a3bbbb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-239.ec2.internal\" (UID: \"f2bc47ecdba8650fc42750ab52a3bbbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.850296 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.850224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81546329125905c90a6831ec0139b393-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"81546329125905c90a6831ec0139b393\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.850819 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.850798    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:27.850918 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.850828    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:27.850918 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.850841    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:27.867958 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.867933    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-239.ec2.internal\" not found" node="ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.871305 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.871290    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-239.ec2.internal\" not found" node="ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.930093 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:27.930069    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:27.950437 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.950408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2bc47ecdba8650fc42750ab52a3bbbb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-239.ec2.internal\" (UID: \"f2bc47ecdba8650fc42750ab52a3bbbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.950548 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.950439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81546329125905c90a6831ec0139b393-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"81546329125905c90a6831ec0139b393\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.950548 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.950464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81546329125905c90a6831ec0139b393-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"81546329125905c90a6831ec0139b393\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.950548 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.950501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2bc47ecdba8650fc42750ab52a3bbbb-config\") pod \"kube-apiserver-proxy-ip-10-0-140-239.ec2.internal\" (UID: \"f2bc47ecdba8650fc42750ab52a3bbbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.950548 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.950511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81546329125905c90a6831ec0139b393-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"81546329125905c90a6831ec0139b393\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:27.950548 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:27.950519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81546329125905c90a6831ec0139b393-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"81546329125905c90a6831ec0139b393\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:28.030355 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:28.030327    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:28.130943 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:28.130863    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:28.170389 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.170350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:28.174163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.174140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:28.232009 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:28.231976    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:28.332494 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:28.332460    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:28.435434 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:28.435379    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:28.535907 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:28.535871    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:28.550904 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.550876    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:28.559600 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.559570    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:28.591946 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.591915    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 13:16:28.592391 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.592042    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:16:28.592391 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.592081    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:16:28.592391 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.592105    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:16:28.636762 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:28.636729    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:28.667951 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.667859    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:11:28 +0000 UTC" deadline="2028-04-07 20:06:34.101372755 +0000 UTC"
Sep 08 13:16:28.667951 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.667893    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13854h50m5.433482964s"
Sep 08 13:16:28.738706 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:28.738672    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:28.839950 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:28.839918    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found"
Sep 08 13:16:28.865709 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.865686    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:28.949376 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.949293    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:28.961405 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.961382    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:28.962428 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.962414    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal"
Sep 08 13:16:28.971705 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:28.971681    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:29.028961 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:29.028929    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81546329125905c90a6831ec0139b393.slice/crio-1f3da58dca280467a0773679401ac8d09ad000f297c3e3fdc84e0941bef9da5f WatchSource:0}: Error finding container 1f3da58dca280467a0773679401ac8d09ad000f297c3e3fdc84e0941bef9da5f: Status 404 returned error can't find the container with id 1f3da58dca280467a0773679401ac8d09ad000f297c3e3fdc84e0941bef9da5f
Sep 08 13:16:29.033801 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.033786    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:16:29.633717 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.633664    2580 apiserver.go:52] "Watching apiserver"
Sep 08 13:16:29.640589 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.640563    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 13:16:29.640898 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.640876    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xjzbp","kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2","openshift-multus/multus-additional-cni-plugins-n86t6","openshift-multus/multus-tswnj","openshift-network-diagnostics/network-check-target-5xdww","openshift-network-operator/iptables-alerter-5rfhl","kube-system/konnectivity-agent-d6qxw","openshift-cluster-node-tuning-operator/tuned-d7rw2","openshift-dns/node-resolver-khkzf","openshift-image-registry/node-ca-4m5ck","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal","openshift-multus/network-metrics-daemon-dbk62","openshift-ovn-kubernetes/ovnkube-node-bd6fv"]
Sep 08 13:16:29.643106 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.643089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.644773 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.644753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:29.644879 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.644820    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:29.645427 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.645410    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:29.645542 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.645446    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-blhbl\""
Sep 08 13:16:29.645542 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.645410    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 13:16:29.645542 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.645529    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:29.646046 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.646030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.647362 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.647342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.647987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.647965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 13:16:29.648165 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.648150    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:29.648626 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.648604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.649163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.649145    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 13:16:29.649264 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.649246    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 13:16:29.650021 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.650005    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:29.650088 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.650054    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:29.650393 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.650369    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 13:16:29.651465 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.651445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.651582 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.651464    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 13:16:29.651582 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.651465    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnkm7\""
Sep 08 13:16:29.651582 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.651503    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4j6gb\""
Sep 08 13:16:29.652320 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.652290    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 13:16:29.652473 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.652332    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 13:16:29.652473 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.652356    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 13:16:29.652473 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.652380    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 13:16:29.652889 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.652870    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4wkmk\""
Sep 08 13:16:29.652974 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.652910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:29.653036 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.652966    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:29.653523 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.653505    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:29.653668 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.653651    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4b7wf\""
Sep 08 13:16:29.653750 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.653651    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:29.654395 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.654377    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.656157 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.656131    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.656618 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.656598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 13:16:29.656709 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.656601    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 13:16:29.656709 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.656651    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 13:16:29.656795 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.656750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9vlj5\""
Sep 08 13:16:29.656924 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.656910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 13:16:29.656967 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.656953    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 13:16:29.657309 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.657295    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 13:16:29.657720 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.657707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.658639 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.658504    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 13:16:29.658639 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.658511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-295rk\""
Sep 08 13:16:29.658639 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.658536    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 13:16:29.658870 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.658854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8b42de1-4263-4bd4-be8e-8b2e65f9da13-host-slash\") pod \"iptables-alerter-5rfhl\" (UID: \"b8b42de1-4263-4bd4-be8e-8b2e65f9da13\") " pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.658940 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.658878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:29.658940 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.658926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/824e0f88-9b13-4a1d-bf08-044dcb419c43-cnibin\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.659034 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.658950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/496cfe43-b060-47e7-a81b-7648af1924f2-cni-binary-copy\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659034 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.658973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-multus-conf-dir\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659034 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.658999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-etc-kubernetes\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659034 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:29.659238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-registration-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.659238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/824e0f88-9b13-4a1d-bf08-044dcb419c43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.659238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/824e0f88-9b13-4a1d-bf08-044dcb419c43-cni-binary-copy\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.659238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkqc\" (UniqueName: \"kubernetes.io/projected/824e0f88-9b13-4a1d-bf08-044dcb419c43-kube-api-access-7xkqc\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.659238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-cnibin\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-multus-socket-dir-parent\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-run-netns\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/496cfe43-b060-47e7-a81b-7648af1924f2-multus-daemon-config\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-run-multus-certs\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7vg\" (UniqueName: \"kubernetes.io/projected/c009591a-887d-4708-a981-bc6005a5a690-kube-api-access-4l7vg\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659377    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-device-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmst5\" (UniqueName: \"kubernetes.io/projected/b828e24c-f36c-4308-b8f8-7225870e2d01-kube-api-access-gmst5\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-hostroot\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-socket-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-system-cni-dir\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-multus-cni-dir\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.659553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhzb\" (UniqueName: \"kubernetes.io/projected/496cfe43-b060-47e7-a81b-7648af1924f2-kube-api-access-jhhzb\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.660060 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8b42de1-4263-4bd4-be8e-8b2e65f9da13-iptables-alerter-script\") pod \"iptables-alerter-5rfhl\" (UID: \"b8b42de1-4263-4bd4-be8e-8b2e65f9da13\") " pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.660060 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/824e0f88-9b13-4a1d-bf08-044dcb419c43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.660060 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.659641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-var-lib-kubelet\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.660343 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.660318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-etc-selinux\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.660603 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.660576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-sys-fs\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.660603 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.660587    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 13:16:29.660754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.660613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/824e0f88-9b13-4a1d-bf08-044dcb419c43-system-cni-dir\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.660754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.660652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-var-lib-cni-multus\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.660754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.660686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9j6f\" (UniqueName: \"kubernetes.io/projected/b8b42de1-4263-4bd4-be8e-8b2e65f9da13-kube-api-access-j9j6f\") pod \"iptables-alerter-5rfhl\" (UID: \"b8b42de1-4263-4bd4-be8e-8b2e65f9da13\") " pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.661366 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.661323    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 13:16:29.661540 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.661365    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 13:16:29.661540 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.661409    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zrsfk\""
Sep 08 13:16:29.661677 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.660717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/824e0f88-9b13-4a1d-bf08-044dcb419c43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.661734 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.661674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-run-k8s-cni-cncf-io\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.661734 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.661708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-var-lib-cni-bin\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.663759 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.663214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 13:16:29.663759 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.663489    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tc4xg\""
Sep 08 13:16:29.663759 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.663532    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 13:16:29.691176 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.691155    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:29.691279 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.691237    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mt5vr"
Sep 08 13:16:29.701712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.701694    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mt5vr"
Sep 08 13:16:29.748733 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.748681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" event={"ID":"81546329125905c90a6831ec0139b393","Type":"ContainerStarted","Data":"1f3da58dca280467a0773679401ac8d09ad000f297c3e3fdc84e0941bef9da5f"}
Sep 08 13:16:29.750364 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.750346    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 13:16:29.762986 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.762957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8a88575-8309-4038-9720-4bc5e427d79c-ovnkube-script-lib\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.763152 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.762994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.763152 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-socket-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.763152 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-system-cni-dir\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.763152 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhzb\" (UniqueName: \"kubernetes.io/projected/496cfe43-b060-47e7-a81b-7648af1924f2-kube-api-access-jhhzb\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.763152 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-sysconfig\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-systemd-units\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8b42de1-4263-4bd4-be8e-8b2e65f9da13-iptables-alerter-script\") pod \"iptables-alerter-5rfhl\" (UID: \"b8b42de1-4263-4bd4-be8e-8b2e65f9da13\") " pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-socket-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-system-cni-dir\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/824e0f88-9b13-4a1d-bf08-044dcb419c43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-var-lib-kubelet\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-kubernetes\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-var-lib-kubelet\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.763375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9377b074-311e-4979-b626-40b1bf8bfccb-dbus\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-slash\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-etc-selinux\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-var-lib-kubelet\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-sys-fs\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/824e0f88-9b13-4a1d-bf08-044dcb419c43-system-cni-dir\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-etc-selinux\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-sys-fs\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-var-lib-cni-multus\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/824e0f88-9b13-4a1d-bf08-044dcb419c43-system-cni-dir\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwshv\" (UniqueName: \"kubernetes.io/projected/aceec167-d19a-4695-8412-94b4c07bea07-kube-api-access-wwshv\") pod \"node-ca-4m5ck\" (UID: \"aceec167-d19a-4695-8412-94b4c07bea07\") " pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-var-lib-cni-multus\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-var-lib-openvswitch\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/824e0f88-9b13-4a1d-bf08-044dcb419c43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.763762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8b42de1-4263-4bd4-be8e-8b2e65f9da13-host-slash\") pod \"iptables-alerter-5rfhl\" (UID: \"b8b42de1-4263-4bd4-be8e-8b2e65f9da13\") " pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/824e0f88-9b13-4a1d-bf08-044dcb419c43-cnibin\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-multus-conf-dir\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8a88575-8309-4038-9720-4bc5e427d79c-ovn-node-metrics-cert\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8b42de1-4263-4bd4-be8e-8b2e65f9da13-host-slash\") pod \"iptables-alerter-5rfhl\" (UID: \"b8b42de1-4263-4bd4-be8e-8b2e65f9da13\") " pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/824e0f88-9b13-4a1d-bf08-044dcb419c43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-registration-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/824e0f88-9b13-4a1d-bf08-044dcb419c43-cnibin\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-sysctl-d\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-multus-conf-dir\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-registration-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.763968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-run\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-run-systemd\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-etc-openvswitch\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8a88575-8309-4038-9720-4bc5e427d79c-env-overrides\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/824e0f88-9b13-4a1d-bf08-044dcb419c43-cni-binary-copy\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkqc\" (UniqueName: \"kubernetes.io/projected/824e0f88-9b13-4a1d-bf08-044dcb419c43-kube-api-access-7xkqc\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.764431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-run-netns\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/496cfe43-b060-47e7-a81b-7648af1924f2-multus-daemon-config\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-run-multus-certs\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/824e0f88-9b13-4a1d-bf08-044dcb419c43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-sys\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-run-netns\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-host\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-run-multus-certs\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmst5\" (UniqueName: \"kubernetes.io/projected/b828e24c-f36c-4308-b8f8-7225870e2d01-kube-api-access-gmst5\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-hostroot\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aceec167-d19a-4695-8412-94b4c07bea07-host\") pod \"node-ca-4m5ck\" (UID: \"aceec167-d19a-4695-8412-94b4c07bea07\") " pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f31cde05-3241-42b0-bf2a-5db1642aab89-konnectivity-ca\") pod \"konnectivity-agent-d6qxw\" (UID: \"f31cde05-3241-42b0-bf2a-5db1642aab89\") " pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-hostroot\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-systemd\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7fz2\" (UniqueName: \"kubernetes.io/projected/49e78f71-60c6-4460-8f7d-8844ef801982-kube-api-access-r7fz2\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-node-log\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-multus-cni-dir\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aceec167-d19a-4695-8412-94b4c07bea07-serviceca\") pod \"node-ca-4m5ck\" (UID: \"aceec167-d19a-4695-8412-94b4c07bea07\") " pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f31cde05-3241-42b0-bf2a-5db1642aab89-agent-certs\") pod \"konnectivity-agent-d6qxw\" (UID: \"f31cde05-3241-42b0-bf2a-5db1642aab89\") " pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-kubelet\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8b42de1-4263-4bd4-be8e-8b2e65f9da13-iptables-alerter-script\") pod \"iptables-alerter-5rfhl\" (UID: \"b8b42de1-4263-4bd4-be8e-8b2e65f9da13\") " pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd49k\" (UniqueName: \"kubernetes.io/projected/e8a88575-8309-4038-9720-4bc5e427d79c-kube-api-access-wd49k\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-multus-cni-dir\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862-hosts-file\") pod \"node-resolver-khkzf\" (UID: \"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862\") " pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49e78f71-60c6-4460-8f7d-8844ef801982-etc-tuned\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-run-netns\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-run-ovn\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-log-socket\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8a88575-8309-4038-9720-4bc5e427d79c-ovnkube-config\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862-tmp-dir\") pod \"node-resolver-khkzf\" (UID: \"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862\") " pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djlbp\" (UniqueName: \"kubernetes.io/projected/c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862-kube-api-access-djlbp\") pod \"node-resolver-khkzf\" (UID: \"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862\") " pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/824e0f88-9b13-4a1d-bf08-044dcb419c43-cni-binary-copy\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.764828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9j6f\" (UniqueName: \"kubernetes.io/projected/b8b42de1-4263-4bd4-be8e-8b2e65f9da13-kube-api-access-j9j6f\") pod \"iptables-alerter-5rfhl\" (UID: \"b8b42de1-4263-4bd4-be8e-8b2e65f9da13\") " pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-run-k8s-cni-cncf-io\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.765712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-var-lib-cni-bin\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-modprobe-d\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/496cfe43-b060-47e7-a81b-7648af1924f2-cni-binary-copy\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-run-k8s-cni-cncf-io\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-etc-kubernetes\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-host-var-lib-cni-bin\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-etc-kubernetes\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/496cfe43-b060-47e7-a81b-7648af1924f2-multus-daemon-config\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.765560    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9377b074-311e-4979-b626-40b1bf8bfccb-kubelet-config\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.765679    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs podName:c009591a-887d-4708-a981-bc6005a5a690 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:30.265657737 +0000 UTC m=+2.965588350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs") pod "network-metrics-daemon-dbk62" (UID: "c009591a-887d-4708-a981-bc6005a5a690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/824e0f88-9b13-4a1d-bf08-044dcb419c43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-sysctl-conf\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-lib-modules\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-run-openvswitch\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.766407 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-cni-bin\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/824e0f88-9b13-4a1d-bf08-044dcb419c43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-cni-netd\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.765866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-cnibin\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.766230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/496cfe43-b060-47e7-a81b-7648af1924f2-cni-binary-copy\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.766346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-cnibin\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.766410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-multus-socket-dir-parent\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.766489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/496cfe43-b060-47e7-a81b-7648af1924f2-multus-socket-dir-parent\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.766514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e78f71-60c6-4460-8f7d-8844ef801982-tmp\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.766549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7vg\" (UniqueName: \"kubernetes.io/projected/c009591a-887d-4708-a981-bc6005a5a690-kube-api-access-4l7vg\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.766584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-device-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.766630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:29.767223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.766664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.769027 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.769002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b828e24c-f36c-4308-b8f8-7225870e2d01-device-dir\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.771153 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.771134    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 13:16:29.775083 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.774895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhzb\" (UniqueName: \"kubernetes.io/projected/496cfe43-b060-47e7-a81b-7648af1924f2-kube-api-access-jhhzb\") pod \"multus-tswnj\" (UID: \"496cfe43-b060-47e7-a81b-7648af1924f2\") " pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.777634 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.777614    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:29.777756 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.777638    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:29.777756 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.777653    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mh4vp for pod openshift-network-diagnostics/network-check-target-5xdww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:29.777756 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.777721    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp podName:708bf191-c98f-4529-be49-f8914698261a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:30.27770377 +0000 UTC m=+2.977634417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mh4vp" (UniqueName: "kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp") pod "network-check-target-5xdww" (UID: "708bf191-c98f-4529-be49-f8914698261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:29.779216 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.779194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9j6f\" (UniqueName: \"kubernetes.io/projected/b8b42de1-4263-4bd4-be8e-8b2e65f9da13-kube-api-access-j9j6f\") pod \"iptables-alerter-5rfhl\" (UID: \"b8b42de1-4263-4bd4-be8e-8b2e65f9da13\") " pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.779550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.779526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmst5\" (UniqueName: \"kubernetes.io/projected/b828e24c-f36c-4308-b8f8-7225870e2d01-kube-api-access-gmst5\") pod \"aws-ebs-csi-driver-node-2thw2\" (UID: \"b828e24c-f36c-4308-b8f8-7225870e2d01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.779801 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.779778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkqc\" (UniqueName: \"kubernetes.io/projected/824e0f88-9b13-4a1d-bf08-044dcb419c43-kube-api-access-7xkqc\") pod \"multus-additional-cni-plugins-n86t6\" (UID: \"824e0f88-9b13-4a1d-bf08-044dcb419c43\") " pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.779943 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.779926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7vg\" (UniqueName: \"kubernetes.io/projected/c009591a-887d-4708-a981-bc6005a5a690-kube-api-access-4l7vg\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:29.867331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e78f71-60c6-4460-8f7d-8844ef801982-tmp\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.867485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:29.867485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.867485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8a88575-8309-4038-9720-4bc5e427d79c-ovnkube-script-lib\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.867485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-sysconfig\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.867485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-sysconfig\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.867485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.867765 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.867495    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:29.867765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-systemd-units\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.867765 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:29.867575    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret podName:9377b074-311e-4979-b626-40b1bf8bfccb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:30.367553905 +0000 UTC m=+3.067484530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret") pod "global-pull-secret-syncer-xjzbp" (UID: "9377b074-311e-4979-b626-40b1bf8bfccb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:29.867765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-systemd-units\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.867765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-kubernetes\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.867765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-var-lib-kubelet\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.867765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9377b074-311e-4979-b626-40b1bf8bfccb-dbus\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:29.867765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-kubernetes\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.867765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-slash\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-slash\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwshv\" (UniqueName: \"kubernetes.io/projected/aceec167-d19a-4695-8412-94b4c07bea07-kube-api-access-wwshv\") pod \"node-ca-4m5ck\" (UID: \"aceec167-d19a-4695-8412-94b4c07bea07\") " pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-var-lib-openvswitch\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-var-lib-kubelet\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8a88575-8309-4038-9720-4bc5e427d79c-ovn-node-metrics-cert\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-var-lib-openvswitch\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9377b074-311e-4979-b626-40b1bf8bfccb-dbus\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-sysctl-d\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-run\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-run-systemd\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.867989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-etc-openvswitch\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8a88575-8309-4038-9720-4bc5e427d79c-env-overrides\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-run-systemd\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-run\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8a88575-8309-4038-9720-4bc5e427d79c-ovnkube-script-lib\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.868168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-sys\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-etc-openvswitch\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-host\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-sysctl-d\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-sys\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aceec167-d19a-4695-8412-94b4c07bea07-host\") pod \"node-ca-4m5ck\" (UID: \"aceec167-d19a-4695-8412-94b4c07bea07\") " pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-host\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f31cde05-3241-42b0-bf2a-5db1642aab89-konnectivity-ca\") pod \"konnectivity-agent-d6qxw\" (UID: \"f31cde05-3241-42b0-bf2a-5db1642aab89\") " pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aceec167-d19a-4695-8412-94b4c07bea07-host\") pod \"node-ca-4m5ck\" (UID: \"aceec167-d19a-4695-8412-94b4c07bea07\") " pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-systemd\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7fz2\" (UniqueName: \"kubernetes.io/projected/49e78f71-60c6-4460-8f7d-8844ef801982-kube-api-access-r7fz2\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-node-log\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-systemd\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aceec167-d19a-4695-8412-94b4c07bea07-serviceca\") pod \"node-ca-4m5ck\" (UID: \"aceec167-d19a-4695-8412-94b4c07bea07\") " pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f31cde05-3241-42b0-bf2a-5db1642aab89-agent-certs\") pod \"konnectivity-agent-d6qxw\" (UID: \"f31cde05-3241-42b0-bf2a-5db1642aab89\") " pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-kubelet\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd49k\" (UniqueName: \"kubernetes.io/projected/e8a88575-8309-4038-9720-4bc5e427d79c-kube-api-access-wd49k\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862-hosts-file\") pod \"node-resolver-khkzf\" (UID: \"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862\") " pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.869135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49e78f71-60c6-4460-8f7d-8844ef801982-etc-tuned\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-run-netns\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-node-log\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-run-ovn\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-log-socket\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8a88575-8309-4038-9720-4bc5e427d79c-ovnkube-config\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862-tmp-dir\") pod \"node-resolver-khkzf\" (UID: \"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862\") " pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djlbp\" (UniqueName: \"kubernetes.io/projected/c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862-kube-api-access-djlbp\") pod \"node-resolver-khkzf\" (UID: \"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862\") " pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-modprobe-d\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9377b074-311e-4979-b626-40b1bf8bfccb-kubelet-config\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-sysctl-conf\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8a88575-8309-4038-9720-4bc5e427d79c-env-overrides\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f31cde05-3241-42b0-bf2a-5db1642aab89-konnectivity-ca\") pod \"konnectivity-agent-d6qxw\" (UID: \"f31cde05-3241-42b0-bf2a-5db1642aab89\") " pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-lib-modules\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-run-ovn\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-run-openvswitch\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-run-netns\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-cni-bin\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.869866 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-cni-netd\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-cni-netd\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-log-socket\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-run-openvswitch\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-cni-bin\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862-hosts-file\") pod \"node-resolver-khkzf\" (UID: \"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862\") " pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862-tmp-dir\") pod \"node-resolver-khkzf\" (UID: \"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862\") " pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9377b074-311e-4979-b626-40b1bf8bfccb-kubelet-config\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-modprobe-d\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8a88575-8309-4038-9720-4bc5e427d79c-ovnkube-config\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a88575-8309-4038-9720-4bc5e427d79c-host-kubelet\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.868721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aceec167-d19a-4695-8412-94b4c07bea07-serviceca\") pod \"node-ca-4m5ck\" (UID: \"aceec167-d19a-4695-8412-94b4c07bea07\") " pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-etc-sysctl-conf\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49e78f71-60c6-4460-8f7d-8844ef801982-lib-modules\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.870422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.869941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e78f71-60c6-4460-8f7d-8844ef801982-tmp\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.870873 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.870513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8a88575-8309-4038-9720-4bc5e427d79c-ovn-node-metrics-cert\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.870938 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.870919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49e78f71-60c6-4460-8f7d-8844ef801982-etc-tuned\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.871217 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.871201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f31cde05-3241-42b0-bf2a-5db1642aab89-agent-certs\") pod \"konnectivity-agent-d6qxw\" (UID: \"f31cde05-3241-42b0-bf2a-5db1642aab89\") " pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:29.880234 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.880208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7fz2\" (UniqueName: \"kubernetes.io/projected/49e78f71-60c6-4460-8f7d-8844ef801982-kube-api-access-r7fz2\") pod \"tuned-d7rw2\" (UID: \"49e78f71-60c6-4460-8f7d-8844ef801982\") " pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.882262 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.882235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djlbp\" (UniqueName: \"kubernetes.io/projected/c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862-kube-api-access-djlbp\") pod \"node-resolver-khkzf\" (UID: \"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862\") " pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.882511 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.882489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd49k\" (UniqueName: \"kubernetes.io/projected/e8a88575-8309-4038-9720-4bc5e427d79c-kube-api-access-wd49k\") pod \"ovnkube-node-bd6fv\" (UID: \"e8a88575-8309-4038-9720-4bc5e427d79c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.882576 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.882560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwshv\" (UniqueName: \"kubernetes.io/projected/aceec167-d19a-4695-8412-94b4c07bea07-kube-api-access-wwshv\") pod \"node-ca-4m5ck\" (UID: \"aceec167-d19a-4695-8412-94b4c07bea07\") " pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:29.955047 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.954956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5rfhl"
Sep 08 13:16:29.963584 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.963565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2"
Sep 08 13:16:29.965274 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:29.965249    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b42de1_4263_4bd4_be8e_8b2e65f9da13.slice/crio-a22c196d1e6afb42f7e1da14d1d6f5aeea755df8bb60c39527ab0395f8eae05d WatchSource:0}: Error finding container a22c196d1e6afb42f7e1da14d1d6f5aeea755df8bb60c39527ab0395f8eae05d: Status 404 returned error can't find the container with id a22c196d1e6afb42f7e1da14d1d6f5aeea755df8bb60c39527ab0395f8eae05d
Sep 08 13:16:29.970793 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.970768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n86t6"
Sep 08 13:16:29.971695 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:29.971675    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb828e24c_f36c_4308_b8f8_7225870e2d01.slice/crio-4b2ddc9bf317b67f674c397fbf430948d609db23d713cf7e2e2feac48a155d89 WatchSource:0}: Error finding container 4b2ddc9bf317b67f674c397fbf430948d609db23d713cf7e2e2feac48a155d89: Status 404 returned error can't find the container with id 4b2ddc9bf317b67f674c397fbf430948d609db23d713cf7e2e2feac48a155d89
Sep 08 13:16:29.975257 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.975242    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tswnj"
Sep 08 13:16:29.979639 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:29.979621    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824e0f88_9b13_4a1d_bf08_044dcb419c43.slice/crio-4533bcd16b153417ce9d1f379a0b4282ef171f9b4cd3eeb681c7235e0397e01a WatchSource:0}: Error finding container 4533bcd16b153417ce9d1f379a0b4282ef171f9b4cd3eeb681c7235e0397e01a: Status 404 returned error can't find the container with id 4533bcd16b153417ce9d1f379a0b4282ef171f9b4cd3eeb681c7235e0397e01a
Sep 08 13:16:29.980484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.980450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7rw2"
Sep 08 13:16:29.985219 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:29.985199    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496cfe43_b060_47e7_a81b_7648af1924f2.slice/crio-1f9ba8ee6db10f9252d3683000ed953215b547421e0ed22562a08de4f894511b WatchSource:0}: Error finding container 1f9ba8ee6db10f9252d3683000ed953215b547421e0ed22562a08de4f894511b: Status 404 returned error can't find the container with id 1f9ba8ee6db10f9252d3683000ed953215b547421e0ed22562a08de4f894511b
Sep 08 13:16:29.986564 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.986489    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:29.992538 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.992520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-khkzf"
Sep 08 13:16:29.993543 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:29.993520    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e78f71_60c6_4460_8f7d_8844ef801982.slice/crio-68e8f8fbf8c6a9087e040752a3c0a3016278fd249b234144baa1347076eee726 WatchSource:0}: Error finding container 68e8f8fbf8c6a9087e040752a3c0a3016278fd249b234144baa1347076eee726: Status 404 returned error can't find the container with id 68e8f8fbf8c6a9087e040752a3c0a3016278fd249b234144baa1347076eee726
Sep 08 13:16:29.996626 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:29.996606    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a88575_8309_4038_9720_4bc5e427d79c.slice/crio-6a63365c7de81dcf9f8f4e7aac948c04298ba827c4e9116e6f1f125c4c549a5e WatchSource:0}: Error finding container 6a63365c7de81dcf9f8f4e7aac948c04298ba827c4e9116e6f1f125c4c549a5e: Status 404 returned error can't find the container with id 6a63365c7de81dcf9f8f4e7aac948c04298ba827c4e9116e6f1f125c4c549a5e
Sep 08 13:16:29.997709 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:29.997690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4m5ck"
Sep 08 13:16:30.001697 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:30.001678    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c65dcc_4c14_4d6d_a71f_07f1ff2e4862.slice/crio-7ce3b2e73db2a8458ad82fcaef67311b10c9f9d0d9f45d0f9bd5fa682f0aa3cb WatchSource:0}: Error finding container 7ce3b2e73db2a8458ad82fcaef67311b10c9f9d0d9f45d0f9bd5fa682f0aa3cb: Status 404 returned error can't find the container with id 7ce3b2e73db2a8458ad82fcaef67311b10c9f9d0d9f45d0f9bd5fa682f0aa3cb
Sep 08 13:16:30.002604 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.002584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:30.006765 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:30.006744    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaceec167_d19a_4695_8412_94b4c07bea07.slice/crio-d1d79b57a5b530c444fad749b52b923797d47d1b071587d38b2a8dbabd596287 WatchSource:0}: Error finding container d1d79b57a5b530c444fad749b52b923797d47d1b071587d38b2a8dbabd596287: Status 404 returned error can't find the container with id d1d79b57a5b530c444fad749b52b923797d47d1b071587d38b2a8dbabd596287
Sep 08 13:16:30.011033 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:30.011011    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31cde05_3241_42b0_bf2a_5db1642aab89.slice/crio-a20a421d7aa5bfe9e8d5c470b931b8648b579fce23f21a78985a22e465305ff0 WatchSource:0}: Error finding container a20a421d7aa5bfe9e8d5c470b931b8648b579fce23f21a78985a22e465305ff0: Status 404 returned error can't find the container with id a20a421d7aa5bfe9e8d5c470b931b8648b579fce23f21a78985a22e465305ff0
Sep 08 13:16:30.186271 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.186236    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:30.206941 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:16:30.206859    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bc47ecdba8650fc42750ab52a3bbbb.slice/crio-e0457b57e34f103c5731759a79d817600a25c2ce727232a49cdf5d5e6e29e8a6 WatchSource:0}: Error finding container e0457b57e34f103c5731759a79d817600a25c2ce727232a49cdf5d5e6e29e8a6: Status 404 returned error can't find the container with id e0457b57e34f103c5731759a79d817600a25c2ce727232a49cdf5d5e6e29e8a6
Sep 08 13:16:30.272011 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.271977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:30.272170 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:30.272148    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:30.272257 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:30.272227    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs podName:c009591a-887d-4708-a981-bc6005a5a690 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:31.272206776 +0000 UTC m=+3.972137408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs") pod "network-metrics-daemon-dbk62" (UID: "c009591a-887d-4708-a981-bc6005a5a690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:30.276420 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.276399    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:30.373366 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.373331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:30.373568 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:30.373524    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:30.373633 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.373571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:30.373633 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:30.373595    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret podName:9377b074-311e-4979-b626-40b1bf8bfccb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:31.373577129 +0000 UTC m=+4.073507757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret") pod "global-pull-secret-syncer-xjzbp" (UID: "9377b074-311e-4979-b626-40b1bf8bfccb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:30.373779 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:30.373714    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:30.373779 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:30.373733    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:30.373779 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:30.373743    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mh4vp for pod openshift-network-diagnostics/network-check-target-5xdww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:30.373901 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:30.373796    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp podName:708bf191-c98f-4529-be49-f8914698261a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:31.373782373 +0000 UTC m=+4.073712992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh4vp" (UniqueName: "kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp") pod "network-check-target-5xdww" (UID: "708bf191-c98f-4529-be49-f8914698261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:30.704351 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.704224    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:30 +0000 UTC" deadline="2028-02-06 17:23:49.895682692 +0000 UTC"
Sep 08 13:16:30.704351 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.704303    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12388h7m19.191384996s"
Sep 08 13:16:30.753544 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.753402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" event={"ID":"f2bc47ecdba8650fc42750ab52a3bbbb","Type":"ContainerStarted","Data":"e0457b57e34f103c5731759a79d817600a25c2ce727232a49cdf5d5e6e29e8a6"}
Sep 08 13:16:30.756644 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.756604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4m5ck" event={"ID":"aceec167-d19a-4695-8412-94b4c07bea07","Type":"ContainerStarted","Data":"d1d79b57a5b530c444fad749b52b923797d47d1b071587d38b2a8dbabd596287"}
Sep 08 13:16:30.760200 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.760091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" event={"ID":"e8a88575-8309-4038-9720-4bc5e427d79c","Type":"ContainerStarted","Data":"6a63365c7de81dcf9f8f4e7aac948c04298ba827c4e9116e6f1f125c4c549a5e"}
Sep 08 13:16:30.765854 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.765822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n86t6" event={"ID":"824e0f88-9b13-4a1d-bf08-044dcb419c43","Type":"ContainerStarted","Data":"4533bcd16b153417ce9d1f379a0b4282ef171f9b4cd3eeb681c7235e0397e01a"}
Sep 08 13:16:30.768589 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.768559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5rfhl" event={"ID":"b8b42de1-4263-4bd4-be8e-8b2e65f9da13","Type":"ContainerStarted","Data":"a22c196d1e6afb42f7e1da14d1d6f5aeea755df8bb60c39527ab0395f8eae05d"}
Sep 08 13:16:30.770357 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.770272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d6qxw" event={"ID":"f31cde05-3241-42b0-bf2a-5db1642aab89","Type":"ContainerStarted","Data":"a20a421d7aa5bfe9e8d5c470b931b8648b579fce23f21a78985a22e465305ff0"}
Sep 08 13:16:30.771758 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.771701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-khkzf" event={"ID":"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862","Type":"ContainerStarted","Data":"7ce3b2e73db2a8458ad82fcaef67311b10c9f9d0d9f45d0f9bd5fa682f0aa3cb"}
Sep 08 13:16:30.773324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.773263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7rw2" event={"ID":"49e78f71-60c6-4460-8f7d-8844ef801982","Type":"ContainerStarted","Data":"68e8f8fbf8c6a9087e040752a3c0a3016278fd249b234144baa1347076eee726"}
Sep 08 13:16:30.775938 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.775860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tswnj" event={"ID":"496cfe43-b060-47e7-a81b-7648af1924f2","Type":"ContainerStarted","Data":"1f9ba8ee6db10f9252d3683000ed953215b547421e0ed22562a08de4f894511b"}
Sep 08 13:16:30.777666 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:30.777585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2" event={"ID":"b828e24c-f36c-4308-b8f8-7225870e2d01","Type":"ContainerStarted","Data":"4b2ddc9bf317b67f674c397fbf430948d609db23d713cf7e2e2feac48a155d89"}
Sep 08 13:16:31.085323 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:31.085237    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:31.280999 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:31.280946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:31.281198 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.281136    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:31.281260 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.281228    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs podName:c009591a-887d-4708-a981-bc6005a5a690 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:33.281203497 +0000 UTC m=+5.981134116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs") pod "network-metrics-daemon-dbk62" (UID: "c009591a-887d-4708-a981-bc6005a5a690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:31.381867 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:31.381720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:31.381867 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:31.381787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:31.382126 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.381910    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:31.382126 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.381938    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:31.382126 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.381953    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mh4vp for pod openshift-network-diagnostics/network-check-target-5xdww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:31.382126 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.382024    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp podName:708bf191-c98f-4529-be49-f8914698261a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:33.38200267 +0000 UTC m=+6.081933289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh4vp" (UniqueName: "kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp") pod "network-check-target-5xdww" (UID: "708bf191-c98f-4529-be49-f8914698261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:31.382126 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.382026    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:31.382126 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.382087    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret podName:9377b074-311e-4979-b626-40b1bf8bfccb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:33.382070548 +0000 UTC m=+6.082001169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret") pod "global-pull-secret-syncer-xjzbp" (UID: "9377b074-311e-4979-b626-40b1bf8bfccb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:31.704653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:31.704498    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:30 +0000 UTC" deadline="2028-05-25 03:00:55.520532492 +0000 UTC"
Sep 08 13:16:31.704653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:31.704543    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14989h44m23.815994656s"
Sep 08 13:16:31.747395 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:31.747254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:31.747654 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.747393    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:31.747654 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:31.747481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:31.747654 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.747597    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:31.747654 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:31.747254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:31.747883 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:31.747705    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:33.299667 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:33.299627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:33.300291 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.299805    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:33.300291 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.299879    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs podName:c009591a-887d-4708-a981-bc6005a5a690 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:37.299857314 +0000 UTC m=+9.999787948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs") pod "network-metrics-daemon-dbk62" (UID: "c009591a-887d-4708-a981-bc6005a5a690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:33.400729 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:33.400680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:33.400913 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:33.400779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:33.400977 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.400919    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:33.400977 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.400940    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:33.400977 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.400953    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mh4vp for pod openshift-network-diagnostics/network-check-target-5xdww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:33.401133 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.401017    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp podName:708bf191-c98f-4529-be49-f8914698261a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:37.400996524 +0000 UTC m=+10.100927160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh4vp" (UniqueName: "kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp") pod "network-check-target-5xdww" (UID: "708bf191-c98f-4529-be49-f8914698261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:33.401591 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.401467    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:33.401591 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.401542    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret podName:9377b074-311e-4979-b626-40b1bf8bfccb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:37.401525122 +0000 UTC m=+10.101455736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret") pod "global-pull-secret-syncer-xjzbp" (UID: "9377b074-311e-4979-b626-40b1bf8bfccb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:33.745005 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:33.744904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:33.745177 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.745041    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:33.745489 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:33.745465    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:33.745603 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.745571    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:33.745722 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:33.745705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:33.745835 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:33.745814    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:35.746288 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:35.745507    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:35.746288 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:35.745628    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:35.746288 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:35.746011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:35.746288 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:35.746107    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:35.746288 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:35.746168    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:35.746288 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:35.746247    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:37.332979 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:37.332944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:37.333457 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.333114    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:37.333457 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.333175    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs podName:c009591a-887d-4708-a981-bc6005a5a690 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:45.333156748 +0000 UTC m=+18.033087364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs") pod "network-metrics-daemon-dbk62" (UID: "c009591a-887d-4708-a981-bc6005a5a690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:37.433309 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:37.433278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:37.433496 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:37.433329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:37.433496 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.433486    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:37.433611 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.433544    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret podName:9377b074-311e-4979-b626-40b1bf8bfccb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:45.43352529 +0000 UTC m=+18.133455917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret") pod "global-pull-secret-syncer-xjzbp" (UID: "9377b074-311e-4979-b626-40b1bf8bfccb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:37.433611 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.433591    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:37.433713 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.433619    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:37.433713 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.433633    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mh4vp for pod openshift-network-diagnostics/network-check-target-5xdww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:37.433713 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.433689    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp podName:708bf191-c98f-4529-be49-f8914698261a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:45.43367106 +0000 UTC m=+18.133601675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh4vp" (UniqueName: "kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp") pod "network-check-target-5xdww" (UID: "708bf191-c98f-4529-be49-f8914698261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:37.748599 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:37.748511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:37.748759 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.748622    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:37.748815 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:37.748755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:37.748869 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.748844    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:37.748922 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:37.748888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:37.749001 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:37.748968    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:39.744900 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:39.744817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:39.744900 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:39.744855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:39.745338 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:39.744823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:39.745338 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:39.744972    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:39.745338 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:39.745059    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:39.745338 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:39.745135    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:41.744857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:41.744825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:41.744857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:41.744868    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:41.745562 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:41.744836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:41.745562 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:41.744969    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:41.745562 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:41.745033    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:41.745562 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:41.745126    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:43.745150 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:43.745064    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:43.745586 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:43.745070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:43.745586 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:43.745188    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:43.745586 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:43.745070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:43.745586 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:43.745304    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:43.745586 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:43.745448    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:45.388826 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:45.388786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:45.389287 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.388955    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:45.389287 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.389025    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs podName:c009591a-887d-4708-a981-bc6005a5a690 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:01.389009211 +0000 UTC m=+34.088939822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs") pod "network-metrics-daemon-dbk62" (UID: "c009591a-887d-4708-a981-bc6005a5a690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:45.489275 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:45.489183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:45.489275 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:45.489225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:45.489499 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.489351    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:45.489499 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.489364    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:45.489499 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.489384    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:45.489499 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.489412    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mh4vp for pod openshift-network-diagnostics/network-check-target-5xdww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:45.489499 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.489422    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret podName:9377b074-311e-4979-b626-40b1bf8bfccb nodeName:}" failed. No retries permitted until 2026-09-08 13:17:01.489391352 +0000 UTC m=+34.189321968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret") pod "global-pull-secret-syncer-xjzbp" (UID: "9377b074-311e-4979-b626-40b1bf8bfccb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:45.489499 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.489454    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp podName:708bf191-c98f-4529-be49-f8914698261a nodeName:}" failed. No retries permitted until 2026-09-08 13:17:01.489441984 +0000 UTC m=+34.189372601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh4vp" (UniqueName: "kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp") pod "network-check-target-5xdww" (UID: "708bf191-c98f-4529-be49-f8914698261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:45.744825 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:45.744739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:45.744825 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:45.744762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:45.744825 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:45.744776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:45.745068 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.744877    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:45.745068 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.744959    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:45.745185 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:45.745076    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:47.746317 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:47.746269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:47.746799 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:47.746386    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:47.746799 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:47.746432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:47.746799 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:47.746491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:47.746799 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:47.746586    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:47.746799 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:47.746632    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:48.824230 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:48.823802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7rw2" event={"ID":"49e78f71-60c6-4460-8f7d-8844ef801982","Type":"ContainerStarted","Data":"6defa593f7a8a073b312785a5f7a15f1606ce9cb16fb7dc25f056a59f5e6c92c"}
Sep 08 13:16:48.826467 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:48.826440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tswnj" event={"ID":"496cfe43-b060-47e7-a81b-7648af1924f2","Type":"ContainerStarted","Data":"d8d030a3ddf4aed45e28f81a0550d3f8fd6ae94dd7623efe3f5e319dc591c806"}
Sep 08 13:16:48.828327 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:48.828175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" event={"ID":"f2bc47ecdba8650fc42750ab52a3bbbb","Type":"ContainerStarted","Data":"c32d5b10a68f97d22449d654b1cc29282d68dbd0fd203960489fe537ebb100ad"}
Sep 08 13:16:48.830691 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:48.830669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" event={"ID":"e8a88575-8309-4038-9720-4bc5e427d79c","Type":"ContainerStarted","Data":"a2ad0b3b66acb94829e0d466105053fc63e3d837020f99c2ec8b648cf1a8fc77"}
Sep 08 13:16:48.830803 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:48.830727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" event={"ID":"e8a88575-8309-4038-9720-4bc5e427d79c","Type":"ContainerStarted","Data":"3394c7918bfa11cbf2eef956a2df745d8d30b9ffa6f041a390feacdb936bf2aa"}
Sep 08 13:16:48.841515 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:48.841475    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d7rw2" podStartSLOduration=2.341588813 podStartE2EDuration="20.841462788s" totalImagesPullingTime="18.499873975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:48.841247714 +0000 UTC m=+21.541178348" watchObservedRunningTime="2026-09-08 13:16:48.841462788 +0000 UTC m=+21.541393421"
Sep 08 13:16:48.881669 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:48.881022    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" podStartSLOduration=19.880998892 podStartE2EDuration="19.880998892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:48.85982035 +0000 UTC m=+21.559751009" watchObservedRunningTime="2026-09-08 13:16:48.880998892 +0000 UTC m=+21.580929530"
Sep 08 13:16:49.745529 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.745495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:49.745529 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.745513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:49.745744 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.745602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:49.745744 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:49.745645    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:49.745744 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:49.745665    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:49.745744 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:49.745739    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:49.834425 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.834378    2580 generic.go:356] "Generic (PLEG): container finished" podID="81546329125905c90a6831ec0139b393" containerID="2bf9018275eccbc79fdf15b38204adb6874c37678841fa9f8a53cec627f11d2a" exitCode=0
Sep 08 13:16:49.835092 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.834428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" event={"ID":"81546329125905c90a6831ec0139b393","Type":"ContainerDied","Data":"2bf9018275eccbc79fdf15b38204adb6874c37678841fa9f8a53cec627f11d2a"}
Sep 08 13:16:49.835963 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.835935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4m5ck" event={"ID":"aceec167-d19a-4695-8412-94b4c07bea07","Type":"ContainerStarted","Data":"340d33d018f5b52e626768d29e5b607a839921b7a3f2f94f2cb45f2324993a56"}
Sep 08 13:16:49.838926 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.838906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" event={"ID":"e8a88575-8309-4038-9720-4bc5e427d79c","Type":"ContainerStarted","Data":"69b4a0fc9ae4a43b8ed829d5b8fc5ed6632c7a0e304dd40bdbf5d4b6aed93a58"}
Sep 08 13:16:49.839027 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.838979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" event={"ID":"e8a88575-8309-4038-9720-4bc5e427d79c","Type":"ContainerStarted","Data":"35f0359d7bf6491e7711d84b08c57fd66f371cec2ad9db635c25628eca5c5e40"}
Sep 08 13:16:49.839076 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.839028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" event={"ID":"e8a88575-8309-4038-9720-4bc5e427d79c","Type":"ContainerStarted","Data":"b75163f90206576c23be00ff6dc5e591e57a2da4924bd9a808ac00bb382c4d08"}
Sep 08 13:16:49.839114 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.839076    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" event={"ID":"e8a88575-8309-4038-9720-4bc5e427d79c","Type":"ContainerStarted","Data":"79f638ee4050161874a62a0af2f3b9c1a0723ce04523cf1bd20da7da6555963b"}
Sep 08 13:16:49.840580 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.840553    2580 generic.go:356] "Generic (PLEG): container finished" podID="824e0f88-9b13-4a1d-bf08-044dcb419c43" containerID="116a702fa7289deefa0935b8dcb14ab65b16e93c309f7c8f368e3218a38b3915" exitCode=0
Sep 08 13:16:49.840674 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.840623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n86t6" event={"ID":"824e0f88-9b13-4a1d-bf08-044dcb419c43","Type":"ContainerDied","Data":"116a702fa7289deefa0935b8dcb14ab65b16e93c309f7c8f368e3218a38b3915"}
Sep 08 13:16:49.841986 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.841959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5rfhl" event={"ID":"b8b42de1-4263-4bd4-be8e-8b2e65f9da13","Type":"ContainerStarted","Data":"399c0dc582c7adda84efb7e1bf90d2b74a220dc7289a96e91b92c6099ab8a063"}
Sep 08 13:16:49.843172 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.843157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d6qxw" event={"ID":"f31cde05-3241-42b0-bf2a-5db1642aab89","Type":"ContainerStarted","Data":"0b01003395216f4c536b6eab5a8da82a0d5e7ea776f8c7106fc91b14ba0358d6"}
Sep 08 13:16:49.844449 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.844428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-khkzf" event={"ID":"c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862","Type":"ContainerStarted","Data":"e99d1785fb17c17bc6977ff9f9ccb2559667b0ba69db1075b4f9f48daf4f17de"}
Sep 08 13:16:49.845753 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.845732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2" event={"ID":"b828e24c-f36c-4308-b8f8-7225870e2d01","Type":"ContainerStarted","Data":"bfd639ec240722ca444469b48f242f369147140d69a4739f7525f7bcf2eca73b"}
Sep 08 13:16:49.850328 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.850290    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tswnj" podStartSLOduration=3.343429123 podStartE2EDuration="21.850278867s" totalImagesPullingTime="18.506849744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:48.882221004 +0000 UTC m=+21.582151630" watchObservedRunningTime="2026-09-08 13:16:49.850278867 +0000 UTC m=+22.550209500"
Sep 08 13:16:49.885569 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.885524    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4m5ck" podStartSLOduration=6.667061963 podStartE2EDuration="21.885510665s" totalImagesPullingTime="15.218448702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:49.885300562 +0000 UTC m=+22.585231196" watchObservedRunningTime="2026-09-08 13:16:49.885510665 +0000 UTC m=+22.585441298"
Sep 08 13:16:49.901120 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.901079    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-khkzf" podStartSLOduration=6.677073524 podStartE2EDuration="21.901065872s" totalImagesPullingTime="15.223992348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:49.900566187 +0000 UTC m=+22.600496838" watchObservedRunningTime="2026-09-08 13:16:49.901065872 +0000 UTC m=+22.600996505"
Sep 08 13:16:49.915411 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:49.915353    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d6qxw" podStartSLOduration=3.432742853 podStartE2EDuration="21.915339164s" totalImagesPullingTime="18.482596311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:49.915249715 +0000 UTC m=+22.615180339" watchObservedRunningTime="2026-09-08 13:16:49.915339164 +0000 UTC m=+22.615269830"
Sep 08 13:16:50.025069 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.025045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:50.025482 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.025464    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:50.035506 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.035480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:50.101835 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.101798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:50.172379 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.172352    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 13:16:50.714663 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.714346    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:16:50.172372488Z","UUID":"d1b1ccdc-f9a7-4b75-adc8-d97f72ea4624","Handler":null,"Name":"","Endpoint":""}
Sep 08 13:16:50.716116 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.716084    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 13:16:50.716116 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.716119    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 13:16:50.850442 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.850364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" event={"ID":"81546329125905c90a6831ec0139b393","Type":"ContainerStarted","Data":"6e385f3a2500e03dd1688e6cb8ca4254645f606f2ee809cea4273d678d02dc09"}
Sep 08 13:16:50.852470 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.852441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2" event={"ID":"b828e24c-f36c-4308-b8f8-7225870e2d01","Type":"ContainerStarted","Data":"964748e50a00782d147f12cffa40069a3594d456c1b2fd37750791beafd0c813"}
Sep 08 13:16:50.852637 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.852618    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:50.853138 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.853119    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d6qxw"
Sep 08 13:16:50.868907 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.868862    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" podStartSLOduration=21.86884653 podStartE2EDuration="21.86884653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:50.868226072 +0000 UTC m=+23.568156705" watchObservedRunningTime="2026-09-08 13:16:50.86884653 +0000 UTC m=+23.568777166"
Sep 08 13:16:50.900616 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.900561    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5rfhl" podStartSLOduration=4.365117661 podStartE2EDuration="22.900543271s" totalImagesPullingTime="18.53542561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:50.900367369 +0000 UTC m=+23.600298004" watchObservedRunningTime="2026-09-08 13:16:50.900543271 +0000 UTC m=+23.600473906"
Sep 08 13:16:50.940141 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:50.940108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:51.021317 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.021217    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:51.106224 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.106188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:51.745692 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.745612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:51.745885 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.745607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:51.745885 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.745607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:51.745885 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:51.745825    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:51.746034 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:51.745731    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:51.746034 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:51.745885    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:51.802496 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.802465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:51.858316 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.858280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" event={"ID":"e8a88575-8309-4038-9720-4bc5e427d79c","Type":"ContainerStarted","Data":"0b1819b72655b54bda92d685b8ce9287fc4dd5f726d9c5c51b2d42a3879efa2b"}
Sep 08 13:16:51.860181 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.860158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2" event={"ID":"b828e24c-f36c-4308-b8f8-7225870e2d01","Type":"ContainerStarted","Data":"134b6dc2d37b83ff0e79cf338cda681fecc24202572a58fe6afe9eb0d3039614"}
Sep 08 13:16:51.880371 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.880326    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2thw2" podStartSLOduration=3.099337003 podStartE2EDuration="23.880312493s" totalImagesPullingTime="20.78097549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:51.879840836 +0000 UTC m=+24.579771471" watchObservedRunningTime="2026-09-08 13:16:51.880312493 +0000 UTC m=+24.580243124"
Sep 08 13:16:51.903502 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:51.903467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:52.634849 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:52.634809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:52.701351 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:52.701313    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:53.748549 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:53.748467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:53.748872 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:53.748471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:53.748872 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:53.748562    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:53.748872 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:53.748478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:53.748872 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:53.748688    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:53.748872 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:53.748727    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:53.867552 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:53.867514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" event={"ID":"e8a88575-8309-4038-9720-4bc5e427d79c","Type":"ContainerStarted","Data":"9213be6b7ef5e60fb7326cd2383d79375ee168cba08ca6c1266e4ed146cbe4b3"}
Sep 08 13:16:53.867802 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:53.867782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:53.867869 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:53.867847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:53.883110 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:53.883085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:53.883236 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:53.883185    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:53.895626 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:53.894363    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv" podStartSLOduration=7.335850975 podStartE2EDuration="25.894344111s" totalImagesPullingTime="18.558493136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:53.893986202 +0000 UTC m=+26.593916837" watchObservedRunningTime="2026-09-08 13:16:53.894344111 +0000 UTC m=+26.594274747"
Sep 08 13:16:54.870800 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:54.870544    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:16:55.424184 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:55.424104    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5xdww"]
Sep 08 13:16:55.424346 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:55.424280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:55.424426 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:55.424383    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:55.427014 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:55.426777    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xjzbp"]
Sep 08 13:16:55.427014 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:55.426882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:55.427014 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:55.426975    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:55.428888 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:55.428658    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dbk62"]
Sep 08 13:16:55.428888 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:55.428760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:55.428888 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:55.428855    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:55.872375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:55.872343    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:16:56.598419 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:56.598323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:16:56.700246 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:56.700211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:56.745486 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:56.745450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:56.745630 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:56.745450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:56.745630 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:56.745556    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:56.745729 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:56.745632    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:56.800883 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:56.800853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:56.872260 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:56.872143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:57.694238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:57.694205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:57.746069 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:57.745748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:57.746069 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:57.745865    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:16:58.745368 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:58.745331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:16:58.745962 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:58.745330    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:16:58.745962 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:58.745479    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:16:58.745962 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:58.745617    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:16:59.745339 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:16:59.745298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:16:59.745534 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:16:59.745438    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xdww" podUID="708bf191-c98f-4529-be49-f8914698261a"
Sep 08 13:17:00.745381 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:00.745347    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:17:00.745666 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:00.745347    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:17:00.745666 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:00.745475    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjzbp" podUID="9377b074-311e-4979-b626-40b1bf8bfccb"
Sep 08 13:17:00.745666 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:00.745527    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:17:01.404082 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.403824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:17:01.404256 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.403980    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:17:01.404256 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.404205    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs podName:c009591a-887d-4708-a981-bc6005a5a690 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:33.404188151 +0000 UTC m=+66.104118770 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs") pod "network-metrics-daemon-dbk62" (UID: "c009591a-887d-4708-a981-bc6005a5a690") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:17:01.504812 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.504783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:17:01.504927 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.504829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:17:01.504969 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.504927    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:17:01.505001 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.504985    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret podName:9377b074-311e-4979-b626-40b1bf8bfccb nodeName:}" failed. No retries permitted until 2026-09-08 13:17:33.504968616 +0000 UTC m=+66.204899243 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret") pod "global-pull-secret-syncer-xjzbp" (UID: "9377b074-311e-4979-b626-40b1bf8bfccb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:17:01.505051 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.504925    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:17:01.505051 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.505025    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:17:01.505051 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.505035    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mh4vp for pod openshift-network-diagnostics/network-check-target-5xdww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:17:01.505142 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.505078    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp podName:708bf191-c98f-4529-be49-f8914698261a nodeName:}" failed. No retries permitted until 2026-09-08 13:17:33.505065374 +0000 UTC m=+66.204995987 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh4vp" (UniqueName: "kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp") pod "network-check-target-5xdww" (UID: "708bf191-c98f-4529-be49-f8914698261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:17:01.539898 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.539872    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeReady"
Sep 08 13:17:01.540012 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.540005    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 13:17:01.642960 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.642915    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tzvsh"]
Sep 08 13:17:01.658322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.658289    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8mnkp"]
Sep 08 13:17:01.658484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.658430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tzvsh"
Sep 08 13:17:01.661153 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.661115    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 13:17:01.661277 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.661116    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 13:17:01.661277 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.661122    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-89flp\""
Sep 08 13:17:01.661277 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.661140    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 13:17:01.672216 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.672194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tzvsh"]
Sep 08 13:17:01.672328 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.672224    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8mnkp"]
Sep 08 13:17:01.672328 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.672314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.675196 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.675173    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 13:17:01.675312 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.675301    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 13:17:01.675390 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.675369    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6zj7\""
Sep 08 13:17:01.745072 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.745039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:17:01.750201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.750174    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7tdz6\""
Sep 08 13:17:01.750201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.750175    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:17:01.750685 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.750213    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:17:01.801425 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.801377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:01.807412 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.807357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc889\" (UniqueName: \"kubernetes.io/projected/e8475a96-b21d-4bd9-9bd7-c1bebed877d2-kube-api-access-sc889\") pod \"ingress-canary-tzvsh\" (UID: \"e8475a96-b21d-4bd9-9bd7-c1bebed877d2\") " pod="openshift-ingress-canary/ingress-canary-tzvsh"
Sep 08 13:17:01.807550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.807418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.807550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.807463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8475a96-b21d-4bd9-9bd7-c1bebed877d2-cert\") pod \"ingress-canary-tzvsh\" (UID: \"e8475a96-b21d-4bd9-9bd7-c1bebed877d2\") " pod="openshift-ingress-canary/ingress-canary-tzvsh"
Sep 08 13:17:01.807550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.807480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdd7\" (UniqueName: \"kubernetes.io/projected/d011fe55-25c5-486c-850f-49d3abbd6650-kube-api-access-zhdd7\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.807550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.807527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d011fe55-25c5-486c-850f-49d3abbd6650-tmp-dir\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.807692 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.807553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d011fe55-25c5-486c-850f-49d3abbd6650-config-volume\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.888071 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.887988    2580 generic.go:356] "Generic (PLEG): container finished" podID="824e0f88-9b13-4a1d-bf08-044dcb419c43" containerID="e09db128c2acd24be970c25081f498bb733046f8780fbc66e89f03cf7e1d7128" exitCode=0
Sep 08 13:17:01.888071 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.888040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n86t6" event={"ID":"824e0f88-9b13-4a1d-bf08-044dcb419c43","Type":"ContainerDied","Data":"e09db128c2acd24be970c25081f498bb733046f8780fbc66e89f03cf7e1d7128"}
Sep 08 13:17:01.908082 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.908050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d011fe55-25c5-486c-850f-49d3abbd6650-tmp-dir\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.908193 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.908097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d011fe55-25c5-486c-850f-49d3abbd6650-config-volume\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.908193 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.908131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc889\" (UniqueName: \"kubernetes.io/projected/e8475a96-b21d-4bd9-9bd7-c1bebed877d2-kube-api-access-sc889\") pod \"ingress-canary-tzvsh\" (UID: \"e8475a96-b21d-4bd9-9bd7-c1bebed877d2\") " pod="openshift-ingress-canary/ingress-canary-tzvsh"
Sep 08 13:17:01.908193 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.908150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.908193 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.908167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8475a96-b21d-4bd9-9bd7-c1bebed877d2-cert\") pod \"ingress-canary-tzvsh\" (UID: \"e8475a96-b21d-4bd9-9bd7-c1bebed877d2\") " pod="openshift-ingress-canary/ingress-canary-tzvsh"
Sep 08 13:17:01.908193 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.908183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdd7\" (UniqueName: \"kubernetes.io/projected/d011fe55-25c5-486c-850f-49d3abbd6650-kube-api-access-zhdd7\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.908459 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.908351    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:01.908459 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:01.908457    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls podName:d011fe55-25c5-486c-850f-49d3abbd6650 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:02.408433435 +0000 UTC m=+35.108364064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls") pod "dns-default-8mnkp" (UID: "d011fe55-25c5-486c-850f-49d3abbd6650") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:01.908545 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.908501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d011fe55-25c5-486c-850f-49d3abbd6650-tmp-dir\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.908727 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.908699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d011fe55-25c5-486c-850f-49d3abbd6650-config-volume\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.912027 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.911999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8475a96-b21d-4bd9-9bd7-c1bebed877d2-cert\") pod \"ingress-canary-tzvsh\" (UID: \"e8475a96-b21d-4bd9-9bd7-c1bebed877d2\") " pod="openshift-ingress-canary/ingress-canary-tzvsh"
Sep 08 13:17:01.928361 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.928334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdd7\" (UniqueName: \"kubernetes.io/projected/d011fe55-25c5-486c-850f-49d3abbd6650-kube-api-access-zhdd7\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:01.941063 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.941027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc889\" (UniqueName: \"kubernetes.io/projected/e8475a96-b21d-4bd9-9bd7-c1bebed877d2-kube-api-access-sc889\") pod \"ingress-canary-tzvsh\" (UID: \"e8475a96-b21d-4bd9-9bd7-c1bebed877d2\") " pod="openshift-ingress-canary/ingress-canary-tzvsh"
Sep 08 13:17:01.961707 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.961676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:01.969473 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:01.969449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tzvsh"
Sep 08 13:17:02.288614 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:02.288519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tzvsh"]
Sep 08 13:17:02.292384 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:17:02.292351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8475a96_b21d_4bd9_9bd7_c1bebed877d2.slice/crio-954cacb1c76682eab86ea6e8ee6b23053fd9d38297468251fdd5183cbeb48cdf WatchSource:0}: Error finding container 954cacb1c76682eab86ea6e8ee6b23053fd9d38297468251fdd5183cbeb48cdf: Status 404 returned error can't find the container with id 954cacb1c76682eab86ea6e8ee6b23053fd9d38297468251fdd5183cbeb48cdf
Sep 08 13:17:02.412524 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:02.412488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:02.412696 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:02.412649    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:02.412743 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:02.412711    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls podName:d011fe55-25c5-486c-850f-49d3abbd6650 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:03.412695228 +0000 UTC m=+36.112625844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls") pod "dns-default-8mnkp" (UID: "d011fe55-25c5-486c-850f-49d3abbd6650") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:02.744851 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:02.744814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:17:02.745010 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:02.744814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:17:02.747312 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:02.747289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:17:02.747485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:02.747291    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:17:02.747838 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:02.747820    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lvcfq\""
Sep 08 13:17:02.891788 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:02.891753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tzvsh" event={"ID":"e8475a96-b21d-4bd9-9bd7-c1bebed877d2","Type":"ContainerStarted","Data":"954cacb1c76682eab86ea6e8ee6b23053fd9d38297468251fdd5183cbeb48cdf"}
Sep 08 13:17:03.420140 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:03.420102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:03.420323 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:03.420272    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:03.420370 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:03.420339    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls podName:d011fe55-25c5-486c-850f-49d3abbd6650 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:05.420322118 +0000 UTC m=+38.120252730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls") pod "dns-default-8mnkp" (UID: "d011fe55-25c5-486c-850f-49d3abbd6650") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:05.433535 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:05.433309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:05.433995 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:05.433510    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:05.433995 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:05.433682    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls podName:d011fe55-25c5-486c-850f-49d3abbd6650 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:09.433657475 +0000 UTC m=+42.133588106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls") pod "dns-default-8mnkp" (UID: "d011fe55-25c5-486c-850f-49d3abbd6650") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:05.901013 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:05.900974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tzvsh" event={"ID":"e8475a96-b21d-4bd9-9bd7-c1bebed877d2","Type":"ContainerStarted","Data":"19ba1047d5c49444e47c1959ddc99d74e62e142d9f53ef2a8cbc15b1f7254455"}
Sep 08 13:17:05.902946 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:05.902919    2580 generic.go:356] "Generic (PLEG): container finished" podID="824e0f88-9b13-4a1d-bf08-044dcb419c43" containerID="00d76a9cec2bc3ce331c98ae73232caf7c7b1c834720fd847177c2f5feea8a72" exitCode=0
Sep 08 13:17:05.903051 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:05.902963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n86t6" event={"ID":"824e0f88-9b13-4a1d-bf08-044dcb419c43","Type":"ContainerDied","Data":"00d76a9cec2bc3ce331c98ae73232caf7c7b1c834720fd847177c2f5feea8a72"}
Sep 08 13:17:05.927761 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:05.927715    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tzvsh" podStartSLOduration=2.388164232 podStartE2EDuration="4.927701064s" totalImagesPullingTime="2.539536832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:05.927519733 +0000 UTC m=+38.627450386" watchObservedRunningTime="2026-09-08 13:17:05.927701064 +0000 UTC m=+38.627631699"
Sep 08 13:17:07.831327 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:07.831283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:07.910500 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:07.910463    2580 generic.go:356] "Generic (PLEG): container finished" podID="824e0f88-9b13-4a1d-bf08-044dcb419c43" containerID="362dbf61397906029b1beb964e4d7077a3442c8e52e2147ddfca919c35e75175" exitCode=0
Sep 08 13:17:07.910694 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:07.910516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n86t6" event={"ID":"824e0f88-9b13-4a1d-bf08-044dcb419c43","Type":"ContainerDied","Data":"362dbf61397906029b1beb964e4d7077a3442c8e52e2147ddfca919c35e75175"}
Sep 08 13:17:08.850411 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:08.850350    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:08.951799 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:08.951759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:09.459818 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:09.459775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:09.459990 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:09.459945    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:09.460030 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:09.460013    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls podName:d011fe55-25c5-486c-850f-49d3abbd6650 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:17.459997522 +0000 UTC m=+50.159928135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls") pod "dns-default-8mnkp" (UID: "d011fe55-25c5-486c-850f-49d3abbd6650") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:11.565472 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:11.565436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:11.624207 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:11.624167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:11.852681 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:11.852641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:11.954924 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:11.954890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:12.925703 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:12.925674    2580 generic.go:356] "Generic (PLEG): container finished" podID="824e0f88-9b13-4a1d-bf08-044dcb419c43" containerID="5a8e7b02193702fc4b9442429e9898720d57000b345a9569aeb5693cd037d2ce" exitCode=0
Sep 08 13:17:12.926070 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:12.925720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n86t6" event={"ID":"824e0f88-9b13-4a1d-bf08-044dcb419c43","Type":"ContainerDied","Data":"5a8e7b02193702fc4b9442429e9898720d57000b345a9569aeb5693cd037d2ce"}
Sep 08 13:17:13.931809 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:13.931776    2580 generic.go:356] "Generic (PLEG): container finished" podID="824e0f88-9b13-4a1d-bf08-044dcb419c43" containerID="943cae632226c7730778e9f80749086b0b32d4b64e22370a3b5868807c23ed0f" exitCode=0
Sep 08 13:17:13.932171 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:13.931830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n86t6" event={"ID":"824e0f88-9b13-4a1d-bf08-044dcb419c43","Type":"ContainerDied","Data":"943cae632226c7730778e9f80749086b0b32d4b64e22370a3b5868807c23ed0f"}
Sep 08 13:17:14.937142 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:14.937110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n86t6" event={"ID":"824e0f88-9b13-4a1d-bf08-044dcb419c43","Type":"ContainerStarted","Data":"78694e389e3455a2982805d2ab49bb02390182cf93c8d9068f9fd9bb0419ec68"}
Sep 08 13:17:14.959142 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:14.959082    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n86t6" podStartSLOduration=12.106056962 podStartE2EDuration="46.959061534s" totalImagesPullingTime="34.853004572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:14.957604957 +0000 UTC m=+47.657535590" watchObservedRunningTime="2026-09-08 13:17:14.959061534 +0000 UTC m=+47.658992167"
Sep 08 13:17:15.554887 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:15.554851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:15.612543 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:15.612508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:17.518480 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:17.518438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:17.519020 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:17.518600    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:17.519020 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:17.518671    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls podName:d011fe55-25c5-486c-850f-49d3abbd6650 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:33.518653398 +0000 UTC m=+66.218584015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls") pod "dns-default-8mnkp" (UID: "d011fe55-25c5-486c-850f-49d3abbd6650") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:26.884660 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:26.884629    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd6fv"
Sep 08 13:17:27.044861 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:27.044830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:27.145499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:27.145387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:33.428573 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.428535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:17:33.430938 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.430918    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:17:33.439108 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:33.439092    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:17:33.439198 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:33.439189    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs podName:c009591a-887d-4708-a981-bc6005a5a690 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:37.43917195 +0000 UTC m=+130.139102562 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs") pod "network-metrics-daemon-dbk62" (UID: "c009591a-887d-4708-a981-bc6005a5a690") : secret "metrics-daemon-secret" not found
Sep 08 13:17:33.529804 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.529768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:17:33.529966 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.529819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:17:33.529966 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.529843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:17:33.529966 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:33.529933    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:33.530125 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:17:33.530012    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls podName:d011fe55-25c5-486c-850f-49d3abbd6650 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:05.529991994 +0000 UTC m=+98.229922626 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls") pod "dns-default-8mnkp" (UID: "d011fe55-25c5-486c-850f-49d3abbd6650") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:33.532331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.532307    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:17:33.532476 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.532339    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:17:33.542657 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.542629    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:17:33.546823 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.546798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9377b074-311e-4979-b626-40b1bf8bfccb-original-pull-secret\") pod \"global-pull-secret-syncer-xjzbp\" (UID: \"9377b074-311e-4979-b626-40b1bf8bfccb\") " pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:17:33.552900 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.552873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh4vp\" (UniqueName: \"kubernetes.io/projected/708bf191-c98f-4529-be49-f8914698261a-kube-api-access-mh4vp\") pod \"network-check-target-5xdww\" (UID: \"708bf191-c98f-4529-be49-f8914698261a\") " pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:17:33.557954 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.557934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7tdz6\""
Sep 08 13:17:33.566741 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.566712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:17:33.662547 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.662513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjzbp"
Sep 08 13:17:33.859749 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.859714    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5xdww"]
Sep 08 13:17:33.863091 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:17:33.863059    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708bf191_c98f_4529_be49_f8914698261a.slice/crio-1c114acdfb3ddfde309cf311daa0c2a47ccfd5a90dd7317f0f5753d4238f10cc WatchSource:0}: Error finding container 1c114acdfb3ddfde309cf311daa0c2a47ccfd5a90dd7317f0f5753d4238f10cc: Status 404 returned error can't find the container with id 1c114acdfb3ddfde309cf311daa0c2a47ccfd5a90dd7317f0f5753d4238f10cc
Sep 08 13:17:33.946700 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.946618    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xjzbp"]
Sep 08 13:17:33.949636 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:17:33.949613    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9377b074_311e_4979_b626_40b1bf8bfccb.slice/crio-63fc33a2ba0066cf5c7cba2fb9223a06641e8f6a071e954521bb83dac4ea62db WatchSource:0}: Error finding container 63fc33a2ba0066cf5c7cba2fb9223a06641e8f6a071e954521bb83dac4ea62db: Status 404 returned error can't find the container with id 63fc33a2ba0066cf5c7cba2fb9223a06641e8f6a071e954521bb83dac4ea62db
Sep 08 13:17:33.982993 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.982960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xjzbp" event={"ID":"9377b074-311e-4979-b626-40b1bf8bfccb","Type":"ContainerStarted","Data":"63fc33a2ba0066cf5c7cba2fb9223a06641e8f6a071e954521bb83dac4ea62db"}
Sep 08 13:17:33.984218 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:33.984195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5xdww" event={"ID":"708bf191-c98f-4529-be49-f8914698261a","Type":"ContainerStarted","Data":"1c114acdfb3ddfde309cf311daa0c2a47ccfd5a90dd7317f0f5753d4238f10cc"}
Sep 08 13:17:34.771919 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:34.771882    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:34.834956 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:34.834916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:37.997189 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:37.997124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xjzbp" event={"ID":"9377b074-311e-4979-b626-40b1bf8bfccb","Type":"ContainerStarted","Data":"4d0e726d7e8531044b145f6d74d6717b5db68356772e098bc1d719f87420f23d"}
Sep 08 13:17:38.014061 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:38.014009    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xjzbp" podStartSLOduration=66.525735222 podStartE2EDuration="1m10.013988181s" totalImagesPullingTime="3.488252959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:38.013357431 +0000 UTC m=+70.713288065" watchObservedRunningTime="2026-09-08 13:17:38.013988181 +0000 UTC m=+70.713918805"
Sep 08 13:17:41.006479 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:41.006444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5xdww" event={"ID":"708bf191-c98f-4529-be49-f8914698261a","Type":"ContainerStarted","Data":"8f89567c8260c2b2590f735966a294e4bfac754d658d509116b6c7b9f4f8fa4b"}
Sep 08 13:17:41.006906 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:41.006580    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:17:41.023843 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:17:41.023793    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5xdww" podStartSLOduration=66.847835206 podStartE2EDuration="1m13.023776557s" totalImagesPullingTime="6.175941351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:41.02293821 +0000 UTC m=+73.722868844" watchObservedRunningTime="2026-09-08 13:17:41.023776557 +0000 UTC m=+73.723707232"
Sep 08 13:18:05.545679 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:18:05.545641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:18:05.546064 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:18:05.545767    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:18:05.546064 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:18:05.545822    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls podName:d011fe55-25c5-486c-850f-49d3abbd6650 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.545807207 +0000 UTC m=+162.245737819 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls") pod "dns-default-8mnkp" (UID: "d011fe55-25c5-486c-850f-49d3abbd6650") : secret "dns-default-metrics-tls" not found
Sep 08 13:18:12.011610 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:18:12.011576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5xdww"
Sep 08 13:18:14.296782 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:18:14.296750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:37.459306 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:18:37.459244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:18:37.459749 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:18:37.459431    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:18:37.459749 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:18:37.459521    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs podName:c009591a-887d-4708-a981-bc6005a5a690 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:39.45950176 +0000 UTC m=+252.159432371 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs") pod "network-metrics-daemon-dbk62" (UID: "c009591a-887d-4708-a981-bc6005a5a690") : secret "metrics-daemon-secret" not found
Sep 08 13:19:01.670258 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.670172    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"]
Sep 08 13:19:01.672140 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.672118    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"]
Sep 08 13:19:01.672264 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.672231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.674229 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.674211    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-b465g"]
Sep 08 13:19:01.674346 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.674328    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:01.675341 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.675321    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 08 13:19:01.675743 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.675725    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 08 13:19:01.675942 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.675912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 08 13:19:01.675942 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.675921    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:19:01.676094 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.675982    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xxcmw\""
Sep 08 13:19:01.676154 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.676114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-b465g"
Sep 08 13:19:01.676468 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.676454    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4xmgr\""
Sep 08 13:19:01.676599 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.676584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:19:01.676948 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.676933    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 08 13:19:01.677011 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.676967    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 08 13:19:01.678985 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.678696    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zlr9g\""
Sep 08 13:19:01.682760 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.682741    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"]
Sep 08 13:19:01.689933 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.689910    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-b465g"]
Sep 08 13:19:01.698757 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.698736    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"]
Sep 08 13:19:01.766751 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.766727    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"]
Sep 08 13:19:01.768765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.768747    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fb85c849f-vl2pw"]
Sep 08 13:19:01.768880 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.768865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:01.771702 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.771681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:01.772379 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.772363    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 13:19:01.772504 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.772370    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 08 13:19:01.773115 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.772877    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 08 13:19:01.773115 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.772944    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 13:19:01.773288 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.773232    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h7gbd\""
Sep 08 13:19:01.776207 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.776185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 13:19:01.776483 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.776457    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 13:19:01.776653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.776558    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 13:19:01.776853 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.776816    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s2n2x\""
Sep 08 13:19:01.781052 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.781034    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 13:19:01.786550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.786526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"]
Sep 08 13:19:01.803405 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.803378    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fb85c849f-vl2pw"]
Sep 08 13:19:01.831982 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.831953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cmvs\" (UniqueName: \"kubernetes.io/projected/4cb6c622-93eb-4efd-b040-d7ba5894398b-kube-api-access-5cmvs\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:01.832101 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.831987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2zvc\" (UniqueName: \"kubernetes.io/projected/704bbd00-297b-4981-ad29-944ecf46aaf0-kube-api-access-l2zvc\") pod \"network-check-source-685456655f-b465g\" (UID: \"704bbd00-297b-4981-ad29-944ecf46aaf0\") " pod="openshift-network-diagnostics/network-check-source-685456655f-b465g"
Sep 08 13:19:01.832101 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.832005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrbt\" (UniqueName: \"kubernetes.io/projected/e515e542-8dc3-4036-8684-bf58e52ee25e-kube-api-access-ktrbt\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:01.832101 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.832071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4cb6c622-93eb-4efd-b040-d7ba5894398b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:01.832201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.832114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:01.832201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.832132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7994c758-9e77-4100-9449-c944e87bf430-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jt8hk\" (UID: \"7994c758-9e77-4100-9449-c944e87bf430\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.832201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.832146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7994c758-9e77-4100-9449-c944e87bf430-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jt8hk\" (UID: \"7994c758-9e77-4100-9449-c944e87bf430\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.832201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.832179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:01.832321 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.832251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjwqp\" (UniqueName: \"kubernetes.io/projected/7994c758-9e77-4100-9449-c944e87bf430-kube-api-access-gjwqp\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jt8hk\" (UID: \"7994c758-9e77-4100-9449-c944e87bf430\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.858386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.858357    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"]
Sep 08 13:19:01.860948 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.860933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:01.863035 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.863009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 08 13:19:01.863035 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.863028    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 08 13:19:01.863414 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.863387    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 08 13:19:01.863616 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.863601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7w8m4\""
Sep 08 13:19:01.863667 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.863622    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:19:01.874036 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.874019    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"]
Sep 08 13:19:01.932786 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:01.932786 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-installation-pull-secrets\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:01.932786 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:01.932786 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-trusted-ca\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjwqp\" (UniqueName: \"kubernetes.io/projected/7994c758-9e77-4100-9449-c944e87bf430-kube-api-access-gjwqp\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jt8hk\" (UID: \"7994c758-9e77-4100-9449-c944e87bf430\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cmvs\" (UniqueName: \"kubernetes.io/projected/4cb6c622-93eb-4efd-b040-d7ba5894398b-kube-api-access-5cmvs\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2zvc\" (UniqueName: \"kubernetes.io/projected/704bbd00-297b-4981-ad29-944ecf46aaf0-kube-api-access-l2zvc\") pod \"network-check-source-685456655f-b465g\" (UID: \"704bbd00-297b-4981-ad29-944ecf46aaf0\") " pod="openshift-network-diagnostics/network-check-source-685456655f-b465g"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrbt\" (UniqueName: \"kubernetes.io/projected/e515e542-8dc3-4036-8684-bf58e52ee25e-kube-api-access-ktrbt\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:01.932867    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-certificates\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88j4\" (UniqueName: \"kubernetes.io/projected/07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8-kube-api-access-n88j4\") pod \"service-ca-operator-859f94c4b-kpj54\" (UID: \"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:01.932935    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls podName:4cb6c622-93eb-4efd-b040-d7ba5894398b nodeName:}" failed. No retries permitted until 2026-09-08 13:19:02.432914937 +0000 UTC m=+155.132845566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tjj2c" (UID: "4cb6c622-93eb-4efd-b040-d7ba5894398b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.932985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8-config\") pod \"service-ca-operator-859f94c4b-kpj54\" (UID: \"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqts\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-kube-api-access-csqts\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4cb6c622-93eb-4efd-b040-d7ba5894398b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-image-registry-private-configuration\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:01.933163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7994c758-9e77-4100-9449-c944e87bf430-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jt8hk\" (UID: \"7994c758-9e77-4100-9449-c944e87bf430\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.933861 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7994c758-9e77-4100-9449-c944e87bf430-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jt8hk\" (UID: \"7994c758-9e77-4100-9449-c944e87bf430\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.933861 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:01.933218    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:19:01.933861 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-bound-sa-token\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:01.933861 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:01.933271    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls podName:e515e542-8dc3-4036-8684-bf58e52ee25e nodeName:}" failed. No retries permitted until 2026-09-08 13:19:02.433256724 +0000 UTC m=+155.133187351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sxmrf" (UID: "e515e542-8dc3-4036-8684-bf58e52ee25e") : secret "samples-operator-tls" not found
Sep 08 13:19:01.933861 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/861463ed-75d5-46a9-8aff-6de821ca7d6d-ca-trust-extracted\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:01.933861 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8-serving-cert\") pod \"service-ca-operator-859f94c4b-kpj54\" (UID: \"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:01.933861 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4cb6c622-93eb-4efd-b040-d7ba5894398b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:01.933861 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.933817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7994c758-9e77-4100-9449-c944e87bf430-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jt8hk\" (UID: \"7994c758-9e77-4100-9449-c944e87bf430\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.935273 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.935257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7994c758-9e77-4100-9449-c944e87bf430-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jt8hk\" (UID: \"7994c758-9e77-4100-9449-c944e87bf430\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.947244 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.947214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cmvs\" (UniqueName: \"kubernetes.io/projected/4cb6c622-93eb-4efd-b040-d7ba5894398b-kube-api-access-5cmvs\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:01.948065 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.948041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjwqp\" (UniqueName: \"kubernetes.io/projected/7994c758-9e77-4100-9449-c944e87bf430-kube-api-access-gjwqp\") pod \"kube-storage-version-migrator-operator-8d7dd4675-jt8hk\" (UID: \"7994c758-9e77-4100-9449-c944e87bf430\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.948324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.948307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrbt\" (UniqueName: \"kubernetes.io/projected/e515e542-8dc3-4036-8684-bf58e52ee25e-kube-api-access-ktrbt\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:01.948453 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.948437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2zvc\" (UniqueName: \"kubernetes.io/projected/704bbd00-297b-4981-ad29-944ecf46aaf0-kube-api-access-l2zvc\") pod \"network-check-source-685456655f-b465g\" (UID: \"704bbd00-297b-4981-ad29-944ecf46aaf0\") " pod="openshift-network-diagnostics/network-check-source-685456655f-b465g"
Sep 08 13:19:01.981997 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.981968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"
Sep 08 13:19:01.993637 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:01.993606    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-b465g"
Sep 08 13:19:02.003238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.003213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.034724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-certificates\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.034768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88j4\" (UniqueName: \"kubernetes.io/projected/07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8-kube-api-access-n88j4\") pod \"service-ca-operator-859f94c4b-kpj54\" (UID: \"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.034798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8-config\") pod \"service-ca-operator-859f94c4b-kpj54\" (UID: \"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.034826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqts\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-kube-api-access-csqts\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.034868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-image-registry-private-configuration\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.034920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-bound-sa-token\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.034967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/861463ed-75d5-46a9-8aff-6de821ca7d6d-ca-trust-extracted\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.034990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8-serving-cert\") pod \"service-ca-operator-859f94c4b-kpj54\" (UID: \"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.035031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-installation-pull-secrets\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.035058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.035499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.035082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-trusted-ca\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.036852 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.036820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-certificates\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.037014 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.036985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/861463ed-75d5-46a9-8aff-6de821ca7d6d-ca-trust-extracted\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.037338 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.037313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-trusted-ca\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.037461 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.037444    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:02.037528 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.037465    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fb85c849f-vl2pw: secret "image-registry-tls" not found
Sep 08 13:19:02.037528 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.037488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8-config\") pod \"service-ca-operator-859f94c4b-kpj54\" (UID: \"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:02.037528 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.037516    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls podName:861463ed-75d5-46a9-8aff-6de821ca7d6d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:02.537498664 +0000 UTC m=+155.237429291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls") pod "image-registry-7fb85c849f-vl2pw" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d") : secret "image-registry-tls" not found
Sep 08 13:19:02.039457 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.039388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-image-registry-private-configuration\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.040084 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.040062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-installation-pull-secrets\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.041086 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.041060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8-serving-cert\") pod \"service-ca-operator-859f94c4b-kpj54\" (UID: \"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:02.045542 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.045517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-bound-sa-token\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.046915 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.046889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqts\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-kube-api-access-csqts\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.046915 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.046903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88j4\" (UniqueName: \"kubernetes.io/projected/07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8-kube-api-access-n88j4\") pod \"service-ca-operator-859f94c4b-kpj54\" (UID: \"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:02.169311 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.169272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"
Sep 08 13:19:02.254409 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.254251    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk"]
Sep 08 13:19:02.257251 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:02.257221    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7994c758_9e77_4100_9449_c944e87bf430.slice/crio-cce3de4fc4572acd55de370ba9756bfc1c7838f8feac1b85dfe7f1546ac6928c WatchSource:0}: Error finding container cce3de4fc4572acd55de370ba9756bfc1c7838f8feac1b85dfe7f1546ac6928c: Status 404 returned error can't find the container with id cce3de4fc4572acd55de370ba9756bfc1c7838f8feac1b85dfe7f1546ac6928c
Sep 08 13:19:02.275235 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.275212    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-b465g"]
Sep 08 13:19:02.277113 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:02.277079    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704bbd00_297b_4981_ad29_944ecf46aaf0.slice/crio-55fabac9a51097859e708704bf5aa23bdd8c0eb443ff353afc91db8482b95a6c WatchSource:0}: Error finding container 55fabac9a51097859e708704bf5aa23bdd8c0eb443ff353afc91db8482b95a6c: Status 404 returned error can't find the container with id 55fabac9a51097859e708704bf5aa23bdd8c0eb443ff353afc91db8482b95a6c
Sep 08 13:19:02.283818 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.283785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:02.436875 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.436834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54"]
Sep 08 13:19:02.438951 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.438927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:02.439062 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.438984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:02.439124 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.439077    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:19:02.439174 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.439124    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:02.439223 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.439137    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls podName:e515e542-8dc3-4036-8684-bf58e52ee25e nodeName:}" failed. No retries permitted until 2026-09-08 13:19:03.439121098 +0000 UTC m=+156.139051723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sxmrf" (UID: "e515e542-8dc3-4036-8684-bf58e52ee25e") : secret "samples-operator-tls" not found
Sep 08 13:19:02.439223 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.439203    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls podName:4cb6c622-93eb-4efd-b040-d7ba5894398b nodeName:}" failed. No retries permitted until 2026-09-08 13:19:03.439184185 +0000 UTC m=+156.139114799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tjj2c" (UID: "4cb6c622-93eb-4efd-b040-d7ba5894398b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:02.439689 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:02.439671    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f2bf17_b1d7_439d_958b_2fb0cb2b8cf8.slice/crio-c2dcf5905751e195e9a516c5f5875f73d664baa2b6256a1cc94e5f31a51e0edc WatchSource:0}: Error finding container c2dcf5905751e195e9a516c5f5875f73d664baa2b6256a1cc94e5f31a51e0edc: Status 404 returned error can't find the container with id c2dcf5905751e195e9a516c5f5875f73d664baa2b6256a1cc94e5f31a51e0edc
Sep 08 13:19:02.539814 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:02.539720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:02.539953 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.539862    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:02.539953 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.539883    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fb85c849f-vl2pw: secret "image-registry-tls" not found
Sep 08 13:19:02.539953 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:02.539938    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls podName:861463ed-75d5-46a9-8aff-6de821ca7d6d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:03.539922045 +0000 UTC m=+156.239852657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls") pod "image-registry-7fb85c849f-vl2pw" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d") : secret "image-registry-tls" not found
Sep 08 13:19:03.200709 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:03.200666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk" event={"ID":"7994c758-9e77-4100-9449-c944e87bf430","Type":"ContainerStarted","Data":"cce3de4fc4572acd55de370ba9756bfc1c7838f8feac1b85dfe7f1546ac6928c"}
Sep 08 13:19:03.202447 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:03.202414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-b465g" event={"ID":"704bbd00-297b-4981-ad29-944ecf46aaf0","Type":"ContainerStarted","Data":"832c56457be516c528035845b5663245f7fd06ad48911343641e3565976c109a"}
Sep 08 13:19:03.202591 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:03.202466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-b465g" event={"ID":"704bbd00-297b-4981-ad29-944ecf46aaf0","Type":"ContainerStarted","Data":"55fabac9a51097859e708704bf5aa23bdd8c0eb443ff353afc91db8482b95a6c"}
Sep 08 13:19:03.203677 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:03.203651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54" event={"ID":"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8","Type":"ContainerStarted","Data":"c2dcf5905751e195e9a516c5f5875f73d664baa2b6256a1cc94e5f31a51e0edc"}
Sep 08 13:19:03.220995 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:03.220897    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-b465g" podStartSLOduration=2.220877076 podStartE2EDuration="2.220877076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:03.22026358 +0000 UTC m=+155.920194216" watchObservedRunningTime="2026-09-08 13:19:03.220877076 +0000 UTC m=+155.920807711"
Sep 08 13:19:03.447724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:03.447684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:03.447899 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:03.447844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:03.447899 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:03.447868    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:03.448016 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:03.447931    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:19:03.448016 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:03.447951    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls podName:4cb6c622-93eb-4efd-b040-d7ba5894398b nodeName:}" failed. No retries permitted until 2026-09-08 13:19:05.447926825 +0000 UTC m=+158.147857444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tjj2c" (UID: "4cb6c622-93eb-4efd-b040-d7ba5894398b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:03.448016 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:03.447976    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls podName:e515e542-8dc3-4036-8684-bf58e52ee25e nodeName:}" failed. No retries permitted until 2026-09-08 13:19:05.447959865 +0000 UTC m=+158.147890487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sxmrf" (UID: "e515e542-8dc3-4036-8684-bf58e52ee25e") : secret "samples-operator-tls" not found
Sep 08 13:19:03.549048 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:03.548963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:03.549182 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:03.549091    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:03.549182 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:03.549104    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fb85c849f-vl2pw: secret "image-registry-tls" not found
Sep 08 13:19:03.549182 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:03.549152    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls podName:861463ed-75d5-46a9-8aff-6de821ca7d6d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:05.549140174 +0000 UTC m=+158.249070787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls") pod "image-registry-7fb85c849f-vl2pw" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d") : secret "image-registry-tls" not found
Sep 08 13:19:04.679920 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:04.679866    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8mnkp" podUID="d011fe55-25c5-486c-850f-49d3abbd6650"
Sep 08 13:19:05.212837 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:05.212744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk" event={"ID":"7994c758-9e77-4100-9449-c944e87bf430","Type":"ContainerStarted","Data":"27da1f0f5ebf7d8281f25f0abbd6a9c5ed890b3b50c251a0a2dc9d7944d37bb4"}
Sep 08 13:19:05.229565 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:05.229503    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk" podStartSLOduration=2.220403101 podStartE2EDuration="4.229482659s" totalImagesPullingTime="2.009079558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:05.228490185 +0000 UTC m=+157.928420820" watchObservedRunningTime="2026-09-08 13:19:05.229482659 +0000 UTC m=+157.929413294"
Sep 08 13:19:05.465880 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:05.465791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:05.465880 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:05.465876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:05.466121 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:05.465958    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:19:05.466121 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:05.465971    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:05.466121 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:05.466108    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls podName:e515e542-8dc3-4036-8684-bf58e52ee25e nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.466085255 +0000 UTC m=+162.166015876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sxmrf" (UID: "e515e542-8dc3-4036-8684-bf58e52ee25e") : secret "samples-operator-tls" not found
Sep 08 13:19:05.466271 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:05.466145    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls podName:4cb6c622-93eb-4efd-b040-d7ba5894398b nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.466127215 +0000 UTC m=+162.166057834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tjj2c" (UID: "4cb6c622-93eb-4efd-b040-d7ba5894398b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:05.566889 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:05.566852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:05.567089 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:05.567033    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:05.567089 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:05.567060    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fb85c849f-vl2pw: secret "image-registry-tls" not found
Sep 08 13:19:05.567196 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:05.567133    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls podName:861463ed-75d5-46a9-8aff-6de821ca7d6d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.56711192 +0000 UTC m=+162.267042541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls") pod "image-registry-7fb85c849f-vl2pw" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d") : secret "image-registry-tls" not found
Sep 08 13:19:05.757286 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:05.757213    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-dbk62" podUID="c009591a-887d-4708-a981-bc6005a5a690"
Sep 08 13:19:06.061822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.061736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:06.176857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.176813    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:06.201028 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.201000    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bdzv9"]
Sep 08 13:19:06.203249 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.203221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.206493 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.206469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 13:19:06.206620 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.206565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-brrfw\""
Sep 08 13:19:06.206847 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.206827    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 13:19:06.206901 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.206858    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 13:19:06.208773 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.208279    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 13:19:06.217793 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.217761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bdzv9"]
Sep 08 13:19:06.220219 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.220191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54" event={"ID":"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8","Type":"ContainerStarted","Data":"ef946799904b1e32114dab657517cf1080871b6b2593134afd2f04f3ccd361e6"}
Sep 08 13:19:06.248163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.248124    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r"]
Sep 08 13:19:06.250806 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.250780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r"
Sep 08 13:19:06.253533 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.253505    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 13:19:06.253533 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.253524    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 13:19:06.253734 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.253510    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w7mv7\""
Sep 08 13:19:06.260821 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.260794    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r"]
Sep 08 13:19:06.263904 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.263854    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54" podStartSLOduration=1.9191801659999999 podStartE2EDuration="5.263834724s" totalImagesPullingTime="3.344654558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:06.262703544 +0000 UTC m=+158.962634178" watchObservedRunningTime="2026-09-08 13:19:06.263834724 +0000 UTC m=+158.963765363"
Sep 08 13:19:06.278571 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.278541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:06.364359 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.364324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:06.374272 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.374236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/269833ab-86c1-409b-957b-9bd903c7ea86-data-volume\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.374372 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.374304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpww2\" (UniqueName: \"kubernetes.io/projected/0d140ce0-20e9-4607-ad37-8c741949a264-kube-api-access-vpww2\") pod \"migrator-fdbb8fdff-fqz9r\" (UID: \"0d140ce0-20e9-4607-ad37-8c741949a264\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r"
Sep 08 13:19:06.374439 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.374416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.374480 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.374464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfknk\" (UniqueName: \"kubernetes.io/projected/269833ab-86c1-409b-957b-9bd903c7ea86-kube-api-access-rfknk\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.374556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.374503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/269833ab-86c1-409b-957b-9bd903c7ea86-crio-socket\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.374556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.374519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.465206 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.465169    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:06.475011 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.474977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/269833ab-86c1-409b-957b-9bd903c7ea86-data-volume\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.475101 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.475038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpww2\" (UniqueName: \"kubernetes.io/projected/0d140ce0-20e9-4607-ad37-8c741949a264-kube-api-access-vpww2\") pod \"migrator-fdbb8fdff-fqz9r\" (UID: \"0d140ce0-20e9-4607-ad37-8c741949a264\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r"
Sep 08 13:19:06.475101 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.475095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.475203 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.475120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfknk\" (UniqueName: \"kubernetes.io/projected/269833ab-86c1-409b-957b-9bd903c7ea86-kube-api-access-rfknk\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.475250 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:06.475215    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:06.475294 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.475247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/269833ab-86c1-409b-957b-9bd903c7ea86-crio-socket\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.475294 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:06.475282    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:06.975263079 +0000 UTC m=+159.675193712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:06.475420 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.475314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/269833ab-86c1-409b-957b-9bd903c7ea86-crio-socket\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.475420 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.475315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.475420 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:06.475355    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:06.475420 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.475381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/269833ab-86c1-409b-957b-9bd903c7ea86-data-volume\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.475616 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:06.475429    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:06.975418393 +0000 UTC m=+159.675349009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:06.483924 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.483901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpww2\" (UniqueName: \"kubernetes.io/projected/0d140ce0-20e9-4607-ad37-8c741949a264-kube-api-access-vpww2\") pod \"migrator-fdbb8fdff-fqz9r\" (UID: \"0d140ce0-20e9-4607-ad37-8c741949a264\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r"
Sep 08 13:19:06.484043 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.484027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfknk\" (UniqueName: \"kubernetes.io/projected/269833ab-86c1-409b-957b-9bd903c7ea86-kube-api-access-rfknk\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.560631 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.560593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r"
Sep 08 13:19:06.834504 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.830780    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r"]
Sep 08 13:19:06.837458 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:06.837430    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d140ce0_20e9_4607_ad37_8c741949a264.slice/crio-3d7289611ab17712ccc4c3aba9ad49ab078272d7076810e56f5d7f72148dea69 WatchSource:0}: Error finding container 3d7289611ab17712ccc4c3aba9ad49ab078272d7076810e56f5d7f72148dea69: Status 404 returned error can't find the container with id 3d7289611ab17712ccc4c3aba9ad49ab078272d7076810e56f5d7f72148dea69
Sep 08 13:19:06.979127 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.979085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.979127 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:06.979135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:06.979339 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:06.979235    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:06.979339 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:06.979239    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:06.979339 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:06.979292    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:07.979276949 +0000 UTC m=+160.679207563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:06.979339 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:06.979309    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:07.979301061 +0000 UTC m=+160.679231676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:07.094851 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:07.094817    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:07.225352 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:07.225316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r" event={"ID":"0d140ce0-20e9-4607-ad37-8c741949a264","Type":"ContainerStarted","Data":"3d7289611ab17712ccc4c3aba9ad49ab078272d7076810e56f5d7f72148dea69"}
Sep 08 13:19:07.987370 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:07.987324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:07.987916 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:07.987377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:07.987916 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:07.987488    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:07.987916 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:07.987560    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.987544449 +0000 UTC m=+162.687475061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:07.987916 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:07.987497    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:07.987916 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:07.987593    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.987587877 +0000 UTC m=+162.687518489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:08.133263 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:08.133234    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-khkzf_c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862/dns-node-resolver/0.log"
Sep 08 13:19:08.932864 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:08.932839    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4m5ck_aceec167-d19a-4695-8412-94b4c07bea07/node-ca/0.log"
Sep 08 13:19:09.400469 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.400414    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-n4pgl"]
Sep 08 13:19:09.403410 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.403380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.405848 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.405831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 13:19:09.405930 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.405869    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 13:19:09.406540 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.406523    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 13:19:09.406621 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.406600    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 13:19:09.406621 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.406609    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zn4lc\""
Sep 08 13:19:09.414340 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.414317    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-n4pgl"]
Sep 08 13:19:09.499999 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.499956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:09.499999 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.500016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4kl\" (UniqueName: \"kubernetes.io/projected/3ec4439f-0c7d-40dd-b995-41b8eb00dca3-kube-api-access-4n4kl\") pod \"service-ca-75d99c484-n4pgl\" (UID: \"3ec4439f-0c7d-40dd-b995-41b8eb00dca3\") " pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.500223 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:09.500117    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:19:09.500223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.500129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ec4439f-0c7d-40dd-b995-41b8eb00dca3-signing-key\") pod \"service-ca-75d99c484-n4pgl\" (UID: \"3ec4439f-0c7d-40dd-b995-41b8eb00dca3\") " pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.500223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.500166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ec4439f-0c7d-40dd-b995-41b8eb00dca3-signing-cabundle\") pod \"service-ca-75d99c484-n4pgl\" (UID: \"3ec4439f-0c7d-40dd-b995-41b8eb00dca3\") " pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.500223 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:09.500180    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls podName:e515e542-8dc3-4036-8684-bf58e52ee25e nodeName:}" failed. No retries permitted until 2026-09-08 13:19:17.50016347 +0000 UTC m=+170.200094081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sxmrf" (UID: "e515e542-8dc3-4036-8684-bf58e52ee25e") : secret "samples-operator-tls" not found
Sep 08 13:19:09.500349 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.500251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:09.500349 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:09.500341    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:09.500449 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:09.500371    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls podName:4cb6c622-93eb-4efd-b040-d7ba5894398b nodeName:}" failed. No retries permitted until 2026-09-08 13:19:17.500363348 +0000 UTC m=+170.200293959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tjj2c" (UID: "4cb6c622-93eb-4efd-b040-d7ba5894398b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:09.600747 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.600704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:09.600909 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.600795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4kl\" (UniqueName: \"kubernetes.io/projected/3ec4439f-0c7d-40dd-b995-41b8eb00dca3-kube-api-access-4n4kl\") pod \"service-ca-75d99c484-n4pgl\" (UID: \"3ec4439f-0c7d-40dd-b995-41b8eb00dca3\") " pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.600909 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.600816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ec4439f-0c7d-40dd-b995-41b8eb00dca3-signing-key\") pod \"service-ca-75d99c484-n4pgl\" (UID: \"3ec4439f-0c7d-40dd-b995-41b8eb00dca3\") " pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.600909 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.600834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ec4439f-0c7d-40dd-b995-41b8eb00dca3-signing-cabundle\") pod \"service-ca-75d99c484-n4pgl\" (UID: \"3ec4439f-0c7d-40dd-b995-41b8eb00dca3\") " pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.600909 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:09.600857    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:09.600909 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.600866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:19:09.600909 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:09.600875    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fb85c849f-vl2pw: secret "image-registry-tls" not found
Sep 08 13:19:09.601136 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:09.600937    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls podName:861463ed-75d5-46a9-8aff-6de821ca7d6d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:17.600918118 +0000 UTC m=+170.300848748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls") pod "image-registry-7fb85c849f-vl2pw" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d") : secret "image-registry-tls" not found
Sep 08 13:19:09.601136 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:09.600972    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:19:09.601136 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:09.601047    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls podName:d011fe55-25c5-486c-850f-49d3abbd6650 nodeName:}" failed. No retries permitted until 2026-09-08 13:21:11.601033059 +0000 UTC m=+284.300963676 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls") pod "dns-default-8mnkp" (UID: "d011fe55-25c5-486c-850f-49d3abbd6650") : secret "dns-default-metrics-tls" not found
Sep 08 13:19:09.601584 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.601559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ec4439f-0c7d-40dd-b995-41b8eb00dca3-signing-cabundle\") pod \"service-ca-75d99c484-n4pgl\" (UID: \"3ec4439f-0c7d-40dd-b995-41b8eb00dca3\") " pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.603272 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.603251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ec4439f-0c7d-40dd-b995-41b8eb00dca3-signing-key\") pod \"service-ca-75d99c484-n4pgl\" (UID: \"3ec4439f-0c7d-40dd-b995-41b8eb00dca3\") " pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.608807 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.608778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4kl\" (UniqueName: \"kubernetes.io/projected/3ec4439f-0c7d-40dd-b995-41b8eb00dca3-kube-api-access-4n4kl\") pod \"service-ca-75d99c484-n4pgl\" (UID: \"3ec4439f-0c7d-40dd-b995-41b8eb00dca3\") " pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.712904 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.712818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-n4pgl"
Sep 08 13:19:09.934559 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.934528    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tzvsh_e8475a96-b21d-4bd9-9bd7-c1bebed877d2/serve-healthcheck-canary/0.log"
Sep 08 13:19:09.989162 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:09.989072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-n4pgl"]
Sep 08 13:19:09.991557 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:09.991533    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec4439f_0c7d_40dd_b995_41b8eb00dca3.slice/crio-a4165288c156fc23fc3aeb265cba9bf8d7398f0b9dbb41d2eff11398e9c1ed45 WatchSource:0}: Error finding container a4165288c156fc23fc3aeb265cba9bf8d7398f0b9dbb41d2eff11398e9c1ed45: Status 404 returned error can't find the container with id a4165288c156fc23fc3aeb265cba9bf8d7398f0b9dbb41d2eff11398e9c1ed45
Sep 08 13:19:10.005230 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:10.005169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:10.005350 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:10.005236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:10.005428 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:10.005358    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:10.005428 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:10.005358    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:10.005523 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:10.005436    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:14.005419828 +0000 UTC m=+166.705350440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:10.005523 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:10.005470    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:14.005451679 +0000 UTC m=+166.705382308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:10.235649 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:10.235605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-n4pgl" event={"ID":"3ec4439f-0c7d-40dd-b995-41b8eb00dca3","Type":"ContainerStarted","Data":"7e2ada59e9adb7407dc086af3c6e40aedeebfe3f8329e6a465916b4de906bafb"}
Sep 08 13:19:10.235835 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:10.235670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-n4pgl" event={"ID":"3ec4439f-0c7d-40dd-b995-41b8eb00dca3","Type":"ContainerStarted","Data":"a4165288c156fc23fc3aeb265cba9bf8d7398f0b9dbb41d2eff11398e9c1ed45"}
Sep 08 13:19:10.606641 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:10.606605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:10.669643 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:10.669612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.222788 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.222733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.244192 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.244153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r" event={"ID":"0d140ce0-20e9-4607-ad37-8c741949a264","Type":"ContainerStarted","Data":"24b1a4ff1d664ce241a17369bac5aa9189383da4766fd55b5a9ec199a8efca44"}
Sep 08 13:19:12.244379 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.244216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r" event={"ID":"0d140ce0-20e9-4607-ad37-8c741949a264","Type":"ContainerStarted","Data":"4a2e0ee355aea852c9db025b67c853fce54f79c4759a494b1b49d4aea5c35fb4"}
Sep 08 13:19:12.263088 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.263019    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fqz9r" podStartSLOduration=1.6527878660000002 podStartE2EDuration="6.262995999s" totalImagesPullingTime="4.610208133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:12.262360763 +0000 UTC m=+164.962291398" watchObservedRunningTime="2026-09-08 13:19:12.262995999 +0000 UTC m=+164.962926636"
Sep 08 13:19:12.263484 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.263450    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-n4pgl" podStartSLOduration=3.263443119 podStartE2EDuration="3.263443119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:10.25622501 +0000 UTC m=+162.956155645" watchObservedRunningTime="2026-09-08 13:19:12.263443119 +0000 UTC m=+164.963373770"
Sep 08 13:19:12.442550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.442517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.543712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.543624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.626817 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.626774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.760267 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.760221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.945501 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:12.945467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:13.775865 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:13.775827    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:13.912525 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:13.912485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.042112 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:14.042010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:14.042112 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:14.042066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:14.042315 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:14.042159    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:14.042315 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:14.042208    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:14.042315 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:14.042223    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:22.042208118 +0000 UTC m=+174.742138730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:14.042315 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:14.042255    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:22.042242562 +0000 UTC m=+174.742173197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:14.131997 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:14.131961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.319207 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:14.319122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.521719 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:14.521683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.712932 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:14.712891    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.912808 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:14.912777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:16.276151 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:16.276098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:16.375157 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:16.375130    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:16.631158 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:16.631127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:16.731913 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:16.731880    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:17.516227 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.516195    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:17.572300 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.572269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:17.572472 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.572351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:17.572472 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:17.572439    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:17.572563 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:17.572497    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls podName:4cb6c622-93eb-4efd-b040-d7ba5894398b nodeName:}" failed. No retries permitted until 2026-09-08 13:19:33.572482439 +0000 UTC m=+186.272413051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tjj2c" (UID: "4cb6c622-93eb-4efd-b040-d7ba5894398b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:17.574728 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.574708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e515e542-8dc3-4036-8684-bf58e52ee25e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sxmrf\" (UID: \"e515e542-8dc3-4036-8684-bf58e52ee25e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:17.587723 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.587701    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"
Sep 08 13:19:17.673237 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.673202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:17.675595 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.675564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") pod \"image-registry-7fb85c849f-vl2pw\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") " pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:17.685562 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.685539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:17.727132 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.727098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:17.746022 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.745991    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:19:17.856663 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.856639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf"]
Sep 08 13:19:17.955409 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:17.955361    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fb85c849f-vl2pw"]
Sep 08 13:19:17.958149 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:17.958119    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861463ed_75d5_46a9_8aff_6de821ca7d6d.slice/crio-f808c68aa0a113e202c0ab399d30f3c33cc76882eb39c0e17ac115dd36e3bd65 WatchSource:0}: Error finding container f808c68aa0a113e202c0ab399d30f3c33cc76882eb39c0e17ac115dd36e3bd65: Status 404 returned error can't find the container with id f808c68aa0a113e202c0ab399d30f3c33cc76882eb39c0e17ac115dd36e3bd65
Sep 08 13:19:18.121378 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:18.121291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:18.263486 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:18.263441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf" event={"ID":"e515e542-8dc3-4036-8684-bf58e52ee25e","Type":"ContainerStarted","Data":"1a845e3a180568e40252e40baa4ab957f1e4bad69d2808a18cc72fbbf93238cf"}
Sep 08 13:19:18.265017 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:18.264984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw" event={"ID":"861463ed-75d5-46a9-8aff-6de821ca7d6d","Type":"ContainerStarted","Data":"aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb"}
Sep 08 13:19:18.265133 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:18.265037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw" event={"ID":"861463ed-75d5-46a9-8aff-6de821ca7d6d","Type":"ContainerStarted","Data":"f808c68aa0a113e202c0ab399d30f3c33cc76882eb39c0e17ac115dd36e3bd65"}
Sep 08 13:19:18.265133 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:18.265125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:18.321204 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:18.321160    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:18.744695 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:18.744658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:19:18.796196 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:18.796156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:18.916331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:18.916296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:19.126803 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:19.126770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:19.411726 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:19.411617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:19.523096 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:19.523061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:20.111771 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:20.111677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:20.274820 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:20.274785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf" event={"ID":"e515e542-8dc3-4036-8684-bf58e52ee25e","Type":"ContainerStarted","Data":"86bed48325cfc1349b881b6e0e471289f6256174cb0200b31a2920a3fee17f1f"}
Sep 08 13:19:20.274982 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:20.274844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf" event={"ID":"e515e542-8dc3-4036-8684-bf58e52ee25e","Type":"ContainerStarted","Data":"7515303ac60462d7b67d4d8953ffb1e676c04563dc61e542533b06c42eb29a35"}
Sep 08 13:19:20.291502 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:20.291460    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw" podStartSLOduration=19.29144324 podStartE2EDuration="19.29144324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:18.286208797 +0000 UTC m=+170.986139433" watchObservedRunningTime="2026-09-08 13:19:20.29144324 +0000 UTC m=+172.991373871"
Sep 08 13:19:20.292298 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:20.292271    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sxmrf" podStartSLOduration=17.676867503 podStartE2EDuration="19.292260697s" totalImagesPullingTime="1.615393194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:20.290874035 +0000 UTC m=+172.990804668" watchObservedRunningTime="2026-09-08 13:19:20.292260697 +0000 UTC m=+172.992191330"
Sep 08 13:19:20.717808 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:20.717775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:20.918513 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:20.918477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:22.111464 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.111423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:22.111838 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.111486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:22.111838 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:22.111503    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:22.111838 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:22.111571    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls podName:269833ab-86c1-409b-957b-9bd903c7ea86 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:38.111552624 +0000 UTC m=+190.811483258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdzv9" (UID: "269833ab-86c1-409b-957b-9bd903c7ea86") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:22.112077 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.112054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/269833ab-86c1-409b-957b-9bd903c7ea86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:22.301960 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.301929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:22.364529 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.364446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:22.514700 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.514670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:22.642124 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.642037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:22.744931 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.744898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:22.826998 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.826963    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:22.920714 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:22.920633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:24.577425 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:24.577373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:32.225037 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:32.224996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:32.300217 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:32.300176    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:33.344755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:33.344723    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fb85c849f-vl2pw"]
Sep 08 13:19:33.348900 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:33.348871    2580 patch_prober.go:28] interesting pod/image-registry-7fb85c849f-vl2pw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 13:19:33.349038 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:33.348921    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw" podUID="861463ed-75d5-46a9-8aff-6de821ca7d6d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:19:33.472790 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:33.472758    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:33.564532 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:33.564495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:33.591255 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:33.591203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:33.593663 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:33.593645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cb6c622-93eb-4efd-b040-d7ba5894398b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tjj2c\" (UID: \"4cb6c622-93eb-4efd-b040-d7ba5894398b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:33.882939 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:33.882907    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h7gbd\""
Sep 08 13:19:33.890997 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:33.890979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"
Sep 08 13:19:34.170630 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:34.170605    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c"]
Sep 08 13:19:34.173345 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:34.173313    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb6c622_93eb_4efd_b040_d7ba5894398b.slice/crio-ad090cc0f9c8a8be6fdc6ca8a9ae479445f8fc40a8890068cca98073176abbbb WatchSource:0}: Error finding container ad090cc0f9c8a8be6fdc6ca8a9ae479445f8fc40a8890068cca98073176abbbb: Status 404 returned error can't find the container with id ad090cc0f9c8a8be6fdc6ca8a9ae479445f8fc40a8890068cca98073176abbbb
Sep 08 13:19:34.320352 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:34.320319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c" event={"ID":"4cb6c622-93eb-4efd-b040-d7ba5894398b","Type":"ContainerStarted","Data":"ad090cc0f9c8a8be6fdc6ca8a9ae479445f8fc40a8890068cca98073176abbbb"}
Sep 08 13:19:37.642325 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:37.642212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:37.743107 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:37.743067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:37.816676 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:37.816633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:37.816856 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:37.816834    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"]
Sep 08 13:19:37.819252 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:37.819235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"
Sep 08 13:19:37.821194 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:37.821172    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 08 13:19:37.821310 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:37.821245    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ntdm4\""
Sep 08 13:19:37.828780 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:37.828762    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"]
Sep 08 13:19:37.923307 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:37.923233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c0a4ef9-eef8-4730-a6c4-ed33c7079eab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-k7rsm\" (UID: \"8c0a4ef9-eef8-4730-a6c4-ed33c7079eab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"
Sep 08 13:19:38.023993 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.023953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c0a4ef9-eef8-4730-a6c4-ed33c7079eab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-k7rsm\" (UID: \"8c0a4ef9-eef8-4730-a6c4-ed33c7079eab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"
Sep 08 13:19:38.024176 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:38.024108    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 13:19:38.024221 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:38.024178    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c0a4ef9-eef8-4730-a6c4-ed33c7079eab-tls-certificates podName:8c0a4ef9-eef8-4730-a6c4-ed33c7079eab nodeName:}" failed. No retries permitted until 2026-09-08 13:19:38.524161719 +0000 UTC m=+191.224092332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8c0a4ef9-eef8-4730-a6c4-ed33c7079eab-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-k7rsm" (UID: "8c0a4ef9-eef8-4730-a6c4-ed33c7079eab") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 13:19:38.124890 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.124860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:38.127188 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.127154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/269833ab-86c1-409b-957b-9bd903c7ea86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdzv9\" (UID: \"269833ab-86c1-409b-957b-9bd903c7ea86\") " pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:38.211090 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.210993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:38.274173 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.274144    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:38.316247 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.316220    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-brrfw\""
Sep 08 13:19:38.324973 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.324955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdzv9"
Sep 08 13:19:38.334498 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.334473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c" event={"ID":"4cb6c622-93eb-4efd-b040-d7ba5894398b","Type":"ContainerStarted","Data":"75aff3009ccf9022d64990c1d700e98eb3fb9223fc02dc7c6101556b27c8fa96"}
Sep 08 13:19:38.353642 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.353600    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tjj2c" podStartSLOduration=34.182000541 podStartE2EDuration="37.35358403s" totalImagesPullingTime="3.171583489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:38.352498415 +0000 UTC m=+191.052429048" watchObservedRunningTime="2026-09-08 13:19:38.35358403 +0000 UTC m=+191.053514663"
Sep 08 13:19:38.529034 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.528933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c0a4ef9-eef8-4730-a6c4-ed33c7079eab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-k7rsm\" (UID: \"8c0a4ef9-eef8-4730-a6c4-ed33c7079eab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"
Sep 08 13:19:38.529175 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:38.529078    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 13:19:38.529175 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:38.529147    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c0a4ef9-eef8-4730-a6c4-ed33c7079eab-tls-certificates podName:8c0a4ef9-eef8-4730-a6c4-ed33c7079eab nodeName:}" failed. No retries permitted until 2026-09-08 13:19:39.52913004 +0000 UTC m=+192.229060651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8c0a4ef9-eef8-4730-a6c4-ed33c7079eab-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-k7rsm" (UID: "8c0a4ef9-eef8-4730-a6c4-ed33c7079eab") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 13:19:38.601051 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.600797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bdzv9"]
Sep 08 13:19:38.603377 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:38.603353    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269833ab_86c1_409b_957b_9bd903c7ea86.slice/crio-140eca2eddf7ceca0d3a030c0ac6b82c5a36ee6b482f32b95a4463dcd2a8c81b WatchSource:0}: Error finding container 140eca2eddf7ceca0d3a030c0ac6b82c5a36ee6b482f32b95a4463dcd2a8c81b: Status 404 returned error can't find the container with id 140eca2eddf7ceca0d3a030c0ac6b82c5a36ee6b482f32b95a4463dcd2a8c81b
Sep 08 13:19:38.763316 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:38.763281    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:39.339609 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:39.339559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdzv9" event={"ID":"269833ab-86c1-409b-957b-9bd903c7ea86","Type":"ContainerStarted","Data":"227f3a065dd90937601da54197ed3991bf19de6761ab0a947fa7af5903ab2662"}
Sep 08 13:19:39.339783 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:39.339649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdzv9" event={"ID":"269833ab-86c1-409b-957b-9bd903c7ea86","Type":"ContainerStarted","Data":"140eca2eddf7ceca0d3a030c0ac6b82c5a36ee6b482f32b95a4463dcd2a8c81b"}
Sep 08 13:19:39.536330 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:39.536277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c0a4ef9-eef8-4730-a6c4-ed33c7079eab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-k7rsm\" (UID: \"8c0a4ef9-eef8-4730-a6c4-ed33c7079eab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"
Sep 08 13:19:39.538685 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:39.538654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c0a4ef9-eef8-4730-a6c4-ed33c7079eab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-k7rsm\" (UID: \"8c0a4ef9-eef8-4730-a6c4-ed33c7079eab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"
Sep 08 13:19:39.630495 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:39.630383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"
Sep 08 13:19:39.898665 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:39.898565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"]
Sep 08 13:19:39.902019 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:39.901984    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0a4ef9_eef8_4730_a6c4_ed33c7079eab.slice/crio-2fc03bf11af50f9d160c57c61dd2fc35a5386b5c567a14f4b7fa5033b4b42fca WatchSource:0}: Error finding container 2fc03bf11af50f9d160c57c61dd2fc35a5386b5c567a14f4b7fa5033b4b42fca: Status 404 returned error can't find the container with id 2fc03bf11af50f9d160c57c61dd2fc35a5386b5c567a14f4b7fa5033b4b42fca
Sep 08 13:19:40.346876 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:40.346799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdzv9" event={"ID":"269833ab-86c1-409b-957b-9bd903c7ea86","Type":"ContainerStarted","Data":"27c519599731abc0f1591a64f14ee6e90bd5dfc6c01f6e99eb0ec1adbf069908"}
Sep 08 13:19:40.348278 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:40.348247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm" event={"ID":"8c0a4ef9-eef8-4730-a6c4-ed33c7079eab","Type":"ContainerStarted","Data":"2fc03bf11af50f9d160c57c61dd2fc35a5386b5c567a14f4b7fa5033b4b42fca"}
Sep 08 13:19:41.353352 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:41.353313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdzv9" event={"ID":"269833ab-86c1-409b-957b-9bd903c7ea86","Type":"ContainerStarted","Data":"8a1a691c11c029e3a10406527c2fa323da8a15b26510568050361380fd9b6d32"}
Sep 08 13:19:41.354769 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:41.354745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm" event={"ID":"8c0a4ef9-eef8-4730-a6c4-ed33c7079eab","Type":"ContainerStarted","Data":"b4512e8640b127e266acd6996e33f9676544ae327969e78b87be9a0be82a0e34"}
Sep 08 13:19:41.354907 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:41.354889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"
Sep 08 13:19:41.359652 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:41.359625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm"
Sep 08 13:19:41.373168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:41.373124    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bdzv9" podStartSLOduration=33.17408492 podStartE2EDuration="35.373108529s" totalImagesPullingTime="2.199023609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:19:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:41.371531633 +0000 UTC m=+194.071462266" watchObservedRunningTime="2026-09-08 13:19:41.373108529 +0000 UTC m=+194.073039162"
Sep 08 13:19:41.387120 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:41.387064    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-k7rsm" podStartSLOduration=3.379518691 podStartE2EDuration="4.387050147s" totalImagesPullingTime="1.007531456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:41.386187206 +0000 UTC m=+194.086117840" watchObservedRunningTime="2026-09-08 13:19:41.387050147 +0000 UTC m=+194.086980781"
Sep 08 13:19:41.852179 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:41.852133    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:41.927715 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:41.927679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:42.414655 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:42.414620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:42.516329 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:42.516292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:42.904320 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:42.904260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:42.968416 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:42.968372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:43.349702 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:43.349671    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:44.409049 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:44.409003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:47.013672 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.013624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:47.243724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.243692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:47.274063 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.273985    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"]
Sep 08 13:19:47.277192 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.277165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.279909 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.279887    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-s97st"]
Sep 08 13:19:47.280802 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.280782    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 08 13:19:47.280986 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.280964    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 13:19:47.281623 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.281503    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t7chs\""
Sep 08 13:19:47.281623 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.281513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 08 13:19:47.283239 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.283219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.285368 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.285350    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 13:19:47.285523 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.285505    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fgxtm\""
Sep 08 13:19:47.286020 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.286000    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 13:19:47.286414 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.286250    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 13:19:47.294864 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.294797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"]
Sep 08 13:19:47.297785 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.297765    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-s97st"]
Sep 08 13:19:47.318795 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.318757    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8q6mk"]
Sep 08 13:19:47.321303 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.321277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.323956 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.323935    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jflq8\""
Sep 08 13:19:47.323956 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.323950    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 13:19:47.324232 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.324214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 13:19:47.324473 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.324455    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 13:19:47.345263 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.345234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:47.395276 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-accelerators-collector-config\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.395439 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8b41c3aa-e580-463a-a272-1facc973a66a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.395439 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b41c3aa-e580-463a-a272-1facc973a66a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.395439 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.395583 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgchb\" (UniqueName: \"kubernetes.io/projected/8b41c3aa-e580-463a-a272-1facc973a66a-kube-api-access-vgchb\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.395583 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-textfile\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.395583 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-wtmp\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.395583 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.395583 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.395755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.395755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.395755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.395755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7896a071-d3fe-4acb-837d-bf0be2356915-metrics-client-ca\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.395755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvbq5\" (UniqueName: \"kubernetes.io/projected/7896a071-d3fe-4acb-837d-bf0be2356915-kube-api-access-kvbq5\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.395755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7896a071-d3fe-4acb-837d-bf0be2356915-sys\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.395954 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.395954 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7896a071-d3fe-4acb-837d-bf0be2356915-root\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.395954 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2fkd\" (UniqueName: \"kubernetes.io/projected/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-kube-api-access-x2fkd\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.395954 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.395890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-tls\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.442002 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.441965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:47.496274 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.496460 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.496460 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.496460 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.496460 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:47.496426    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 08 13:19:47.496661 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:47.496499    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-tls podName:8b41c3aa-e580-463a-a272-1facc973a66a nodeName:}" failed. No retries permitted until 2026-09-08 13:19:47.996478744 +0000 UTC m=+200.696409379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-s97st" (UID: "8b41c3aa-e580-463a-a272-1facc973a66a") : secret "kube-state-metrics-tls" not found
Sep 08 13:19:47.496739 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7896a071-d3fe-4acb-837d-bf0be2356915-metrics-client-ca\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.496798 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvbq5\" (UniqueName: \"kubernetes.io/projected/7896a071-d3fe-4acb-837d-bf0be2356915-kube-api-access-kvbq5\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.496851 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7896a071-d3fe-4acb-837d-bf0be2356915-sys\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.496851 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.496949 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7896a071-d3fe-4acb-837d-bf0be2356915-root\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.496949 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2fkd\" (UniqueName: \"kubernetes.io/projected/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-kube-api-access-x2fkd\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.497046 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.496986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-tls\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.497046 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-accelerators-collector-config\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.497140 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7896a071-d3fe-4acb-837d-bf0be2356915-metrics-client-ca\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.497140 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8b41c3aa-e580-463a-a272-1facc973a66a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.497140 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b41c3aa-e580-463a-a272-1facc973a66a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.497140 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.497331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.497331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgchb\" (UniqueName: \"kubernetes.io/projected/8b41c3aa-e580-463a-a272-1facc973a66a-kube-api-access-vgchb\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.497331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-textfile\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.497331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-wtmp\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.497331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.497597 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8b41c3aa-e580-463a-a272-1facc973a66a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.497597 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7896a071-d3fe-4acb-837d-bf0be2356915-root\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.498431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.497948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.498431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.498087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7896a071-d3fe-4acb-837d-bf0be2356915-sys\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.498431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.498102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-accelerators-collector-config\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.498431 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:47.498140    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 08 13:19:47.498431 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:47.498204    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-openshift-state-metrics-tls podName:fb556705-9f11-4fe6-aa5a-ad7fd14743ec nodeName:}" failed. No retries permitted until 2026-09-08 13:19:47.998186844 +0000 UTC m=+200.698117459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-bdc9h" (UID: "fb556705-9f11-4fe6-aa5a-ad7fd14743ec") : secret "openshift-state-metrics-tls" not found
Sep 08 13:19:47.498431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.498324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-wtmp\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.498431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.498346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-textfile\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.498850 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.498662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b41c3aa-e580-463a-a272-1facc973a66a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.501075 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.501053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.501465 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.501442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.501860 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.501834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.517941 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.517910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvbq5\" (UniqueName: \"kubernetes.io/projected/7896a071-d3fe-4acb-837d-bf0be2356915-kube-api-access-kvbq5\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.518050 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.517992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2fkd\" (UniqueName: \"kubernetes.io/projected/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-kube-api-access-x2fkd\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:47.518050 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.517991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgchb\" (UniqueName: \"kubernetes.io/projected/8b41c3aa-e580-463a-a272-1facc973a66a-kube-api-access-vgchb\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:47.518428 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.518390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7896a071-d3fe-4acb-837d-bf0be2356915-node-exporter-tls\") pod \"node-exporter-8q6mk\" (UID: \"7896a071-d3fe-4acb-837d-bf0be2356915\") " pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.542542 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.542466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:47.631706 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.631678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8q6mk"
Sep 08 13:19:47.641938 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:47.641907    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7896a071_d3fe_4acb_837d_bf0be2356915.slice/crio-31ac988eaad731387206378071fc82fb3f18814a233c61216982884f58e3c107 WatchSource:0}: Error finding container 31ac988eaad731387206378071fc82fb3f18814a233c61216982884f58e3c107: Status 404 returned error can't find the container with id 31ac988eaad731387206378071fc82fb3f18814a233c61216982884f58e3c107
Sep 08 13:19:47.645632 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.645604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:47.745569 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.745518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:47.985703 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:47.985665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:48.001714 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.001685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:48.001853 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.001727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:48.004167 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.004137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41c3aa-e580-463a-a272-1facc973a66a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-s97st\" (UID: \"8b41c3aa-e580-463a-a272-1facc973a66a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:48.004270 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.004209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb556705-9f11-4fe6-aa5a-ad7fd14743ec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-bdc9h\" (UID: \"fb556705-9f11-4fe6-aa5a-ad7fd14743ec\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:48.190341 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.190304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"
Sep 08 13:19:48.197534 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.197508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st"
Sep 08 13:19:48.323019 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.322984    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:48.380843 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.380779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8q6mk" event={"ID":"7896a071-d3fe-4acb-837d-bf0be2356915","Type":"ContainerStarted","Data":"31ac988eaad731387206378071fc82fb3f18814a233c61216982884f58e3c107"}
Sep 08 13:19:48.485629 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.485598    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h"]
Sep 08 13:19:48.503278 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.503250    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-s97st"]
Sep 08 13:19:48.509178 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.509148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:48.521599 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:48.521570    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb556705_9f11_4fe6_aa5a_ad7fd14743ec.slice/crio-abbe14aee5e936ab7dd5dfec621906d8365230bdec13ab4615fa6652ee50eee8 WatchSource:0}: Error finding container abbe14aee5e936ab7dd5dfec621906d8365230bdec13ab4615fa6652ee50eee8: Status 404 returned error can't find the container with id abbe14aee5e936ab7dd5dfec621906d8365230bdec13ab4615fa6652ee50eee8
Sep 08 13:19:48.522115 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:48.522094    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b41c3aa_e580_463a_a272_1facc973a66a.slice/crio-5b93cf0b943ad8f19beb97f2c09362767bf2fcfd86448dd511c581fb3d62f2cb WatchSource:0}: Error finding container 5b93cf0b943ad8f19beb97f2c09362767bf2fcfd86448dd511c581fb3d62f2cb: Status 404 returned error can't find the container with id 5b93cf0b943ad8f19beb97f2c09362767bf2fcfd86448dd511c581fb3d62f2cb
Sep 08 13:19:48.640430 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:48.640329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:49.386475 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:49.386434    2580 generic.go:356] "Generic (PLEG): container finished" podID="7896a071-d3fe-4acb-837d-bf0be2356915" containerID="4a3d67a9a60aef07e0a0b91197e5158e72d1f0f89c0208a747c9d53042373611" exitCode=0
Sep 08 13:19:49.386933 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:49.386511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8q6mk" event={"ID":"7896a071-d3fe-4acb-837d-bf0be2356915","Type":"ContainerDied","Data":"4a3d67a9a60aef07e0a0b91197e5158e72d1f0f89c0208a747c9d53042373611"}
Sep 08 13:19:49.388252 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:49.388226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st" event={"ID":"8b41c3aa-e580-463a-a272-1facc973a66a","Type":"ContainerStarted","Data":"5b93cf0b943ad8f19beb97f2c09362767bf2fcfd86448dd511c581fb3d62f2cb"}
Sep 08 13:19:49.390563 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:49.390524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h" event={"ID":"fb556705-9f11-4fe6-aa5a-ad7fd14743ec","Type":"ContainerStarted","Data":"be927bff15b60a32613692f2773774dd1bb25cf318041e16f19522730ace7b90"}
Sep 08 13:19:49.390659 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:49.390581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h" event={"ID":"fb556705-9f11-4fe6-aa5a-ad7fd14743ec","Type":"ContainerStarted","Data":"1b9df924d146ffddae9184801d799acc76c721ef20b789394a0753ef5bae2aca"}
Sep 08 13:19:49.390659 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:49.390610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h" event={"ID":"fb556705-9f11-4fe6-aa5a-ad7fd14743ec","Type":"ContainerStarted","Data":"abbe14aee5e936ab7dd5dfec621906d8365230bdec13ab4615fa6652ee50eee8"}
Sep 08 13:19:50.290864 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.290761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:50.390880 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.390831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:50.397537 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.397505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8q6mk" event={"ID":"7896a071-d3fe-4acb-837d-bf0be2356915","Type":"ContainerStarted","Data":"0850a59d0f7b98555babfc567dc9458ec23bf0d4bb552dd10941522bcc1d4092"}
Sep 08 13:19:50.397690 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.397589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8q6mk" event={"ID":"7896a071-d3fe-4acb-837d-bf0be2356915","Type":"ContainerStarted","Data":"fdeb25fa4d619391fce0033fd980f6f11faaf8c253d139403d4dc27db0d5ad61"}
Sep 08 13:19:50.400646 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.400528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st" event={"ID":"8b41c3aa-e580-463a-a272-1facc973a66a","Type":"ContainerStarted","Data":"3091dd309a1fe8608bb50410fa984f5298dacd0e866a96edeb2b4ab4e4142a4e"}
Sep 08 13:19:50.400646 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.400584    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st" event={"ID":"8b41c3aa-e580-463a-a272-1facc973a66a","Type":"ContainerStarted","Data":"622252faa879125dd5a0fdfe42034fb001c35647b3ccba5eaf2e0d20d02758fb"}
Sep 08 13:19:50.400646 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.400612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st" event={"ID":"8b41c3aa-e580-463a-a272-1facc973a66a","Type":"ContainerStarted","Data":"a84ebcce6e9c3888b62d1d13ae1d43d7927b477154973b26ba61c1b31acd219a"}
Sep 08 13:19:50.418674 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.418621    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8q6mk" podStartSLOduration=2.435684648 podStartE2EDuration="3.418602594s" totalImagesPullingTime="905.257632ms" totalInitContainerRuntime="77.660314ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:50.416716992 +0000 UTC m=+203.116647639" watchObservedRunningTime="2026-09-08 13:19:50.418602594 +0000 UTC m=+203.118533226"
Sep 08 13:19:50.437460 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.437412    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-s97st" podStartSLOduration=2.191599355 podStartE2EDuration="3.437379245s" totalImagesPullingTime="1.24577989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:50.436044601 +0000 UTC m=+203.135975254" watchObservedRunningTime="2026-09-08 13:19:50.437379245 +0000 UTC m=+203.137310093"
Sep 08 13:19:50.646621 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:50.646591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:51.409832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.409785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h" event={"ID":"fb556705-9f11-4fe6-aa5a-ad7fd14743ec","Type":"ContainerStarted","Data":"6d8b06fdc22e97c881f6256e426f72e54917dc2605b5b6d7abac7d60b36f2164"}
Sep 08 13:19:51.429831 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.429786    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-bdc9h" podStartSLOduration=2.508517869 podStartE2EDuration="4.429772305s" totalImagesPullingTime="1.921254436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:51.428118666 +0000 UTC m=+204.128049300" watchObservedRunningTime="2026-09-08 13:19:51.429772305 +0000 UTC m=+204.129702938"
Sep 08 13:19:51.530195 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.530163    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"]
Sep 08 13:19:51.532434 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.532417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.535371 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.535167    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8zglb\""
Sep 08 13:19:51.535371 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.535184    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 13:19:51.535575 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.535504    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 08 13:19:51.535684 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.535666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7j300ajq0t0n\""
Sep 08 13:19:51.535810 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.535764    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 08 13:19:51.535888 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.535872    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 08 13:19:51.545446 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.545424    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"]
Sep 08 13:19:51.552427 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.552388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:51.636145 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.636106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/649d85bf-146e-4b48-a20c-82e20d920117-secret-metrics-server-tls\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.636292 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.636164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/649d85bf-146e-4b48-a20c-82e20d920117-client-ca-bundle\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.636292 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.636277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/649d85bf-146e-4b48-a20c-82e20d920117-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.636353 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.636303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpzrm\" (UniqueName: \"kubernetes.io/projected/649d85bf-146e-4b48-a20c-82e20d920117-kube-api-access-fpzrm\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.636353 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.636328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/649d85bf-146e-4b48-a20c-82e20d920117-secret-metrics-server-client-certs\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.636471 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.636349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/649d85bf-146e-4b48-a20c-82e20d920117-audit-log\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.636471 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.636386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/649d85bf-146e-4b48-a20c-82e20d920117-metrics-server-audit-profiles\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.737307 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.737220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/649d85bf-146e-4b48-a20c-82e20d920117-metrics-server-audit-profiles\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.737307 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.737266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/649d85bf-146e-4b48-a20c-82e20d920117-secret-metrics-server-tls\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.737307 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.737300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/649d85bf-146e-4b48-a20c-82e20d920117-client-ca-bundle\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.737594 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.737340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/649d85bf-146e-4b48-a20c-82e20d920117-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.737594 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.737355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpzrm\" (UniqueName: \"kubernetes.io/projected/649d85bf-146e-4b48-a20c-82e20d920117-kube-api-access-fpzrm\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.737594 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.737383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/649d85bf-146e-4b48-a20c-82e20d920117-secret-metrics-server-client-certs\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.737594 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.737418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/649d85bf-146e-4b48-a20c-82e20d920117-audit-log\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.737832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.737808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/649d85bf-146e-4b48-a20c-82e20d920117-audit-log\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.738129 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.738102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/649d85bf-146e-4b48-a20c-82e20d920117-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.738432 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.738413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/649d85bf-146e-4b48-a20c-82e20d920117-metrics-server-audit-profiles\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.739916 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.739891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/649d85bf-146e-4b48-a20c-82e20d920117-secret-metrics-server-tls\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.740020 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.739955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/649d85bf-146e-4b48-a20c-82e20d920117-client-ca-bundle\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.740094 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.740068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/649d85bf-146e-4b48-a20c-82e20d920117-secret-metrics-server-client-certs\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.745221 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.745197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpzrm\" (UniqueName: \"kubernetes.io/projected/649d85bf-146e-4b48-a20c-82e20d920117-kube-api-access-fpzrm\") pod \"metrics-server-7d9bd9bbd9-rkz2x\" (UID: \"649d85bf-146e-4b48-a20c-82e20d920117\") " pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.842260 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.842227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:19:51.989897 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:51.989808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:52.065559 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:52.065521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:52.112041 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:52.112010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"]
Sep 08 13:19:52.114878 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:52.114856    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649d85bf_146e_4b48_a20c_82e20d920117.slice/crio-14afc81c8aa5f144e1ddda2f47c3e2a9df7d4e9991a497830692209388f25ddc WatchSource:0}: Error finding container 14afc81c8aa5f144e1ddda2f47c3e2a9df7d4e9991a497830692209388f25ddc: Status 404 returned error can't find the container with id 14afc81c8aa5f144e1ddda2f47c3e2a9df7d4e9991a497830692209388f25ddc
Sep 08 13:19:52.207837 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:52.207809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:52.415499 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:52.415467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x" event={"ID":"649d85bf-146e-4b48-a20c-82e20d920117","Type":"ContainerStarted","Data":"14afc81c8aa5f144e1ddda2f47c3e2a9df7d4e9991a497830692209388f25ddc"}
Sep 08 13:19:52.669613 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:52.669528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:53.598522 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.598490    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:19:53.601354 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.601333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.604488 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.604467    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b8ad66i9d596i\""
Sep 08 13:19:53.604882 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.604864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 13:19:53.605364 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.605342    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 13:19:53.605487 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.605378    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 13:19:53.605487 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.605378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 13:19:53.605487 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.605476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 13:19:53.605693 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.605678    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 13:19:53.610272 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.610235    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 13:19:53.610367 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.610280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 13:19:53.610457 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.610378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qm8xt\""
Sep 08 13:19:53.610621 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.610607    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 13:19:53.610936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.610910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 13:19:53.611024 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.611007    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 13:19:53.611079 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.611025    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 13:19:53.613125 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.613108    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 13:19:53.618705 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.618687    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 13:19:53.631150 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.628054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:19:53.651033 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.650997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651033 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651234 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651234 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651234 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651234 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651424 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651424 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651424 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651424 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scv5m\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-kube-api-access-scv5m\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651424 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-config-out\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-web-config\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-config\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651653 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651966 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.651966 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.651685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.752666 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-web-config\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.752666 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.752666 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.752933 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-config\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.752933 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.752933 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.752933 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.752933 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.752933 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.752970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scv5m\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-kube-api-access-scv5m\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-config-out\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753211 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.753862 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.753767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.754384 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.754358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.754493 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.754428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.755583 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.755553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.756592 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.755793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.756592 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.756455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.756773 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.756748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-web-config\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.756831 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.756753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-config\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.757533 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.757210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.757533 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.757465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.758444 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.758391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.758580 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.758555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.759044 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.759023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.759130 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.759108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.759346 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.759327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-config-out\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.759981 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.759961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.760548 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.760529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.760917 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.760895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.763456 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.763435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scv5m\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-kube-api-access-scv5m\") pod \"prometheus-k8s-0\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:53.857897 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.857857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:53.911451 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:53.911414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:19:54.196541 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:54.196515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:19:54.198556 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:19:54.198533    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe32dd25_593c_46b1_a683_27582846133b.slice/crio-2deab6357ec484935185574c7d76d4ca3eee08b6fc7490af3919e7a6a9d2d26e WatchSource:0}: Error finding container 2deab6357ec484935185574c7d76d4ca3eee08b6fc7490af3919e7a6a9d2d26e: Status 404 returned error can't find the container with id 2deab6357ec484935185574c7d76d4ca3eee08b6fc7490af3919e7a6a9d2d26e
Sep 08 13:19:54.424552 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:54.424512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x" event={"ID":"649d85bf-146e-4b48-a20c-82e20d920117","Type":"ContainerStarted","Data":"a0438c6be5e30be932cf291dbd956d99089b0b0a76ecd9ba5762f620f6a45a54"}
Sep 08 13:19:54.425754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:54.425728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerStarted","Data":"2deab6357ec484935185574c7d76d4ca3eee08b6fc7490af3919e7a6a9d2d26e"}
Sep 08 13:19:54.497643 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:54.497546    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x" podStartSLOduration=2.142911535 podStartE2EDuration="3.497531657s" totalImagesPullingTime="1.354620122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:54.496407458 +0000 UTC m=+207.196338088" watchObservedRunningTime="2026-09-08 13:19:54.497531657 +0000 UTC m=+207.197462302"
Sep 08 13:19:54.796769 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:54.796686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:55.431243 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:55.431158    2580 generic.go:356] "Generic (PLEG): container finished" podID="be32dd25-593c-46b1-a683-27582846133b" containerID="e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def" exitCode=0
Sep 08 13:19:55.431381 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:55.431247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerDied","Data":"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"}
Sep 08 13:19:56.205980 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:56.205944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:56.277535 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:56.277495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:56.355312 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:56.355275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.078317 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:57.078282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.180061 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:57.180025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.275126 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:57.275091    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.373544 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:57.373517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.563741 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:57.563710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.749350 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:57.749251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.822554 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:57.822524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.979553 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:57.979523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:58.061370 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.061274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:58.183938 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.183902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:58.255044 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.255001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:58.364026 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.363989    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw" podUID="861463ed-75d5-46a9-8aff-6de821ca7d6d" containerName="registry" containerID="cri-o://aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb" gracePeriod=30
Sep 08 13:19:58.609744 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.609720    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:58.697970 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.697883    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") pod \"861463ed-75d5-46a9-8aff-6de821ca7d6d\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") "
Sep 08 13:19:58.697970 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.697925    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-certificates\") pod \"861463ed-75d5-46a9-8aff-6de821ca7d6d\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") "
Sep 08 13:19:58.697970 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.697963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-trusted-ca\") pod \"861463ed-75d5-46a9-8aff-6de821ca7d6d\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") "
Sep 08 13:19:58.698227 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.697980    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-bound-sa-token\") pod \"861463ed-75d5-46a9-8aff-6de821ca7d6d\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") "
Sep 08 13:19:58.698227 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.698013    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/861463ed-75d5-46a9-8aff-6de821ca7d6d-ca-trust-extracted\") pod \"861463ed-75d5-46a9-8aff-6de821ca7d6d\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") "
Sep 08 13:19:58.698227 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.698041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-image-registry-private-configuration\") pod \"861463ed-75d5-46a9-8aff-6de821ca7d6d\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") "
Sep 08 13:19:58.698227 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.698082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csqts\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-kube-api-access-csqts\") pod \"861463ed-75d5-46a9-8aff-6de821ca7d6d\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") "
Sep 08 13:19:58.698227 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.698127    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-installation-pull-secrets\") pod \"861463ed-75d5-46a9-8aff-6de821ca7d6d\" (UID: \"861463ed-75d5-46a9-8aff-6de821ca7d6d\") "
Sep 08 13:19:58.698502 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.698478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "861463ed-75d5-46a9-8aff-6de821ca7d6d" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:58.698830 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.698787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "861463ed-75d5-46a9-8aff-6de821ca7d6d" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:58.700898 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.700870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "861463ed-75d5-46a9-8aff-6de821ca7d6d" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:58.701004 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.700869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-kube-api-access-csqts" (OuterVolumeSpecName: "kube-api-access-csqts") pod "861463ed-75d5-46a9-8aff-6de821ca7d6d" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d"). InnerVolumeSpecName "kube-api-access-csqts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:58.701004 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.700987    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "861463ed-75d5-46a9-8aff-6de821ca7d6d" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:58.701077 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.700998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "861463ed-75d5-46a9-8aff-6de821ca7d6d" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:58.701077 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.701018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "861463ed-75d5-46a9-8aff-6de821ca7d6d" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:58.706875 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.706855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861463ed-75d5-46a9-8aff-6de821ca7d6d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "861463ed-75d5-46a9-8aff-6de821ca7d6d" (UID: "861463ed-75d5-46a9-8aff-6de821ca7d6d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:19:58.755612 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.755580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:58.799792 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.799766    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:19:58.799792 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.799791    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-registry-certificates\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:19:58.799926 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.799801    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/861463ed-75d5-46a9-8aff-6de821ca7d6d-trusted-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:19:58.799926 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.799809    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-bound-sa-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:19:58.799926 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.799818    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/861463ed-75d5-46a9-8aff-6de821ca7d6d-ca-trust-extracted\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:19:58.799926 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.799826    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-image-registry-private-configuration\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:19:58.799926 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.799836    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csqts\" (UniqueName: \"kubernetes.io/projected/861463ed-75d5-46a9-8aff-6de821ca7d6d-kube-api-access-csqts\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:19:58.799926 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.799845    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/861463ed-75d5-46a9-8aff-6de821ca7d6d-installation-pull-secrets\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:19:58.966844 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:58.966760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:59.453050 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.453008    2580 generic.go:356] "Generic (PLEG): container finished" podID="861463ed-75d5-46a9-8aff-6de821ca7d6d" containerID="aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb" exitCode=0
Sep 08 13:19:59.453519 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.453059    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw"
Sep 08 13:19:59.453519 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.453060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw" event={"ID":"861463ed-75d5-46a9-8aff-6de821ca7d6d","Type":"ContainerDied","Data":"aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb"}
Sep 08 13:19:59.453519 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.453176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fb85c849f-vl2pw" event={"ID":"861463ed-75d5-46a9-8aff-6de821ca7d6d","Type":"ContainerDied","Data":"f808c68aa0a113e202c0ab399d30f3c33cc76882eb39c0e17ac115dd36e3bd65"}
Sep 08 13:19:59.453519 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.453202    2580 scope.go:117] "RemoveContainer" containerID="aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb"
Sep 08 13:19:59.463124 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.463079    2580 scope.go:117] "RemoveContainer" containerID="aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb"
Sep 08 13:19:59.463617 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:19:59.463467    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb\": container with ID starting with aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb not found: ID does not exist" containerID="aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb"
Sep 08 13:19:59.463617 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.463496    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb"} err="failed to get container status \"aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb\": rpc error: code = NotFound desc = could not find container \"aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb\": container with ID starting with aa99c94d4ac20dd917da99efa251a1e4b532f796e30f9d41fbdd338c015a59cb not found: ID does not exist"
Sep 08 13:19:59.477200 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.477178    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fb85c849f-vl2pw"]
Sep 08 13:19:59.480780 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.480757    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fb85c849f-vl2pw"]
Sep 08 13:19:59.485711 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.485688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:59.751390 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:19:59.751300    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="861463ed-75d5-46a9-8aff-6de821ca7d6d" path="/var/lib/kubelet/pods/861463ed-75d5-46a9-8aff-6de821ca7d6d/volumes"
Sep 08 13:20:00.185106 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:00.185069    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:00.289804 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:00.289771    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:02.469384 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:02.469353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerStarted","Data":"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"}
Sep 08 13:20:02.469782 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:02.469451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerStarted","Data":"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"}
Sep 08 13:20:04.482327 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:04.482295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerStarted","Data":"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"}
Sep 08 13:20:04.482824 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:04.482359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerStarted","Data":"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"}
Sep 08 13:20:04.482824 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:04.482386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerStarted","Data":"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"}
Sep 08 13:20:04.482824 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:04.482426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerStarted","Data":"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"}
Sep 08 13:20:04.517345 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:04.517278    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5067449 podStartE2EDuration="11.517256191s" totalImagesPullingTime="8.986708703s" totalInitContainerRuntime="23.802588ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:04.515077642 +0000 UTC m=+217.215008275" watchObservedRunningTime="2026-09-08 13:20:04.517256191 +0000 UTC m=+217.217186827"
Sep 08 13:20:08.912259 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:08.912216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:09.219223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:09.219143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:09.280995 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:09.280960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:10.504447 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:10.504335    2580 generic.go:356] "Generic (PLEG): container finished" podID="7994c758-9e77-4100-9449-c944e87bf430" containerID="27da1f0f5ebf7d8281f25f0abbd6a9c5ed890b3b50c251a0a2dc9d7944d37bb4" exitCode=0
Sep 08 13:20:10.504447 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:10.504378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk" event={"ID":"7994c758-9e77-4100-9449-c944e87bf430","Type":"ContainerDied","Data":"27da1f0f5ebf7d8281f25f0abbd6a9c5ed890b3b50c251a0a2dc9d7944d37bb4"}
Sep 08 13:20:10.504867 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:10.504743    2580 scope.go:117] "RemoveContainer" containerID="27da1f0f5ebf7d8281f25f0abbd6a9c5ed890b3b50c251a0a2dc9d7944d37bb4"
Sep 08 13:20:11.510573 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:11.510538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-jt8hk" event={"ID":"7994c758-9e77-4100-9449-c944e87bf430","Type":"ContainerStarted","Data":"f8403d273bfcc12b89e6be801c2f65a182cb501ef9e0a86117b8a2d93df8d007"}
Sep 08 13:20:11.842378 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:11.842347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:20:11.842551 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:11.842428    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:20:12.597349 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:12.597301    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:20.003644 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:20.003606    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:22.646523 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:22.646476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:22.747497 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:22.747457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:24.935236 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:24.935201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:25.035552 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:25.035522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:26.514020 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:26.513988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:26.577271 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:26.577231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:30.412963 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:30.412926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:30.485781 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:30.485744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:31.847738 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:31.847707    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:20:31.851585 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:31.851554    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d9bd9bbd9-rkz2x"
Sep 08 13:20:35.736116 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:35.736072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:35.833114 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:35.833075    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:35.934621 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:35.934592    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:36.607294 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:36.607266    2580 generic.go:356] "Generic (PLEG): container finished" podID="07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8" containerID="ef946799904b1e32114dab657517cf1080871b6b2593134afd2f04f3ccd361e6" exitCode=0
Sep 08 13:20:36.607465 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:36.607340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54" event={"ID":"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8","Type":"ContainerDied","Data":"ef946799904b1e32114dab657517cf1080871b6b2593134afd2f04f3ccd361e6"}
Sep 08 13:20:36.607682 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:36.607669    2580 scope.go:117] "RemoveContainer" containerID="ef946799904b1e32114dab657517cf1080871b6b2593134afd2f04f3ccd361e6"
Sep 08 13:20:37.612538 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:37.612505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-kpj54" event={"ID":"07f2bf17-b1d7-439d-958b-2fb0cb2b8cf8","Type":"ContainerStarted","Data":"9c19ab47843ad250f5963ed183d0640ff08b03d6bc055b2432d419f56e36b643"}
Sep 08 13:20:39.539377 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:39.539326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:20:39.541883 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:39.541846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c009591a-887d-4708-a981-bc6005a5a690-metrics-certs\") pod \"network-metrics-daemon-dbk62\" (UID: \"c009591a-887d-4708-a981-bc6005a5a690\") " pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:20:39.748113 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:39.748086    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lvcfq\""
Sep 08 13:20:39.756521 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:39.756494    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbk62"
Sep 08 13:20:40.026418 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:40.026374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dbk62"]
Sep 08 13:20:40.029065 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:20:40.029042    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc009591a_887d_4708_a981_bc6005a5a690.slice/crio-c5fe206f46fd7bb212f63240cabfcb307b5a507be6dbfad643ac0e418fca8964 WatchSource:0}: Error finding container c5fe206f46fd7bb212f63240cabfcb307b5a507be6dbfad643ac0e418fca8964: Status 404 returned error can't find the container with id c5fe206f46fd7bb212f63240cabfcb307b5a507be6dbfad643ac0e418fca8964
Sep 08 13:20:40.624261 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:40.624225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbk62" event={"ID":"c009591a-887d-4708-a981-bc6005a5a690","Type":"ContainerStarted","Data":"c5fe206f46fd7bb212f63240cabfcb307b5a507be6dbfad643ac0e418fca8964"}
Sep 08 13:20:44.644010 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:44.643973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbk62" event={"ID":"c009591a-887d-4708-a981-bc6005a5a690","Type":"ContainerStarted","Data":"101fd94f48f3ee220cea7a602a1f96b57ae908e2b26cbb0dd6c091841d31ea88"}
Sep 08 13:20:44.644458 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:44.644026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbk62" event={"ID":"c009591a-887d-4708-a981-bc6005a5a690","Type":"ContainerStarted","Data":"bde9d0f36df422b897236e01ffa2c9b9b82d1ff2e809dbe38e4d2f86d7a41a58"}
Sep 08 13:20:44.661285 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:44.661194    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dbk62" podStartSLOduration=252.676640901 podStartE2EDuration="4m16.661176638s" totalImagesPullingTime="3.984535737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:44.660189284 +0000 UTC m=+257.360119918" watchObservedRunningTime="2026-09-08 13:20:44.661176638 +0000 UTC m=+257.361107271"
Sep 08 13:20:44.777488 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:44.777451    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:46.924552 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:46.924517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:46.985687 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:46.985648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:51.636391 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:51.636317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:51.700940 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:51.700906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:52.639120 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:52.639086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:52.740904 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:52.740864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:53.911989 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:53.911960    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:53.931196 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:53.931164    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:54.696331 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:20:54.696301    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:06.561164 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:06.561122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:06.675646 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:06.675607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:11.619094 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.619036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:21:11.621426 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.621389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d011fe55-25c5-486c-850f-49d3abbd6650-metrics-tls\") pod \"dns-default-8mnkp\" (UID: \"d011fe55-25c5-486c-850f-49d3abbd6650\") " pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:21:11.748454 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.748421    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6zj7\""
Sep 08 13:21:11.757243 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.757214    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:21:11.991040 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.990957    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:11.991282 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.991242    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="prometheus" containerID="cri-o://19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2" gracePeriod=600
Sep 08 13:21:11.991364 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.991253    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy" containerID="cri-o://51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328" gracePeriod=600
Sep 08 13:21:11.991364 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.991312    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="config-reloader" containerID="cri-o://92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c" gracePeriod=600
Sep 08 13:21:11.991364 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.991323    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy-web" containerID="cri-o://f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f" gracePeriod=600
Sep 08 13:21:11.991638 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.991281    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219" gracePeriod=600
Sep 08 13:21:11.991638 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:11.991313    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="thanos-sidecar" containerID="cri-o://c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7" gracePeriod=600
Sep 08 13:21:12.027158 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.027134    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8mnkp"]
Sep 08 13:21:12.030826 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:21:12.030795    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd011fe55_25c5_486c_850f_49d3abbd6650.slice/crio-471ecf3aa4ca9665a4878bbde4ca6a214ec86391535ac66407793d8771ceb558 WatchSource:0}: Error finding container 471ecf3aa4ca9665a4878bbde4ca6a214ec86391535ac66407793d8771ceb558: Status 404 returned error can't find the container with id 471ecf3aa4ca9665a4878bbde4ca6a214ec86391535ac66407793d8771ceb558
Sep 08 13:21:12.223084 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.223049    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.326070 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.325994    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-trusted-ca-bundle\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326070 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scv5m\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-kube-api-access-scv5m\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326070 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-thanos-prometheus-http-client-file\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-web-config\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-0\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-2\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326138    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-config-out\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-kube-rbac-proxy\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-grpc-tls\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-1\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326288    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-config\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-metrics-client-certs\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-tls-assets\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-tls\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-db\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326489    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-serving-certs-ca-bundle\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-kubelet-serving-ca-bundle\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326562    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-metrics-client-ca\") pod \"be32dd25-593c-46b1-a683-27582846133b\" (UID: \"be32dd25-593c-46b1-a683-27582846133b\") "
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326618    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326918    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.326936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.326939    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.328552 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.328226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:21:12.328917 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.328887    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:12.329518 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.329078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:12.329518 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.329208    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:12.329518 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.329216    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:12.329518 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.329269    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-config-out" (OuterVolumeSpecName: "config-out") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:21:12.329518 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.329385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:12.330127 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.330062    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:12.330560 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.330529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:12.330728 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.330694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-kube-api-access-scv5m" (OuterVolumeSpecName: "kube-api-access-scv5m") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "kube-api-access-scv5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:21:12.330728 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.330710    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:12.330943 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.330831    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-config" (OuterVolumeSpecName: "config") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:12.330943 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.330860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:12.331117 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.331093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:12.331481 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.331464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:12.331646 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.331632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:21:12.332495 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.332473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:12.340462 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.340436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-web-config" (OuterVolumeSpecName: "web-config") pod "be32dd25-593c-46b1-a683-27582846133b" (UID: "be32dd25-593c-46b1-a683-27582846133b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:12.427892 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427863    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.427892 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427891    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-metrics-client-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427902    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-tls-assets\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427911    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427921    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427930    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-db\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427938    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427947    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427956    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-configmap-metrics-client-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427964    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427972    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scv5m\" (UniqueName: \"kubernetes.io/projected/be32dd25-593c-46b1-a683-27582846133b-kube-api-access-scv5m\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427980    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427988    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-web-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.427995    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be32dd25-593c-46b1-a683-27582846133b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.428002    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be32dd25-593c-46b1-a683-27582846133b-config-out\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.428010    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-kube-rbac-proxy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.428020    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.428066 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.428027    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be32dd25-593c-46b1-a683-27582846133b-secret-grpc-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:21:12.754105 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754072    2580 generic.go:356] "Generic (PLEG): container finished" podID="be32dd25-593c-46b1-a683-27582846133b" containerID="34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219" exitCode=0
Sep 08 13:21:12.754105 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754098    2580 generic.go:356] "Generic (PLEG): container finished" podID="be32dd25-593c-46b1-a683-27582846133b" containerID="51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328" exitCode=0
Sep 08 13:21:12.754105 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754104    2580 generic.go:356] "Generic (PLEG): container finished" podID="be32dd25-593c-46b1-a683-27582846133b" containerID="f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f" exitCode=0
Sep 08 13:21:12.754105 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754110    2580 generic.go:356] "Generic (PLEG): container finished" podID="be32dd25-593c-46b1-a683-27582846133b" containerID="c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7" exitCode=0
Sep 08 13:21:12.754105 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754115    2580 generic.go:356] "Generic (PLEG): container finished" podID="be32dd25-593c-46b1-a683-27582846133b" containerID="92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c" exitCode=0
Sep 08 13:21:12.754105 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754120    2580 generic.go:356] "Generic (PLEG): container finished" podID="be32dd25-593c-46b1-a683-27582846133b" containerID="19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2" exitCode=0
Sep 08 13:21:12.754822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerDied","Data":"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"}
Sep 08 13:21:12.754822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754170    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.754822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerDied","Data":"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"}
Sep 08 13:21:12.754822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerDied","Data":"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"}
Sep 08 13:21:12.754822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerDied","Data":"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"}
Sep 08 13:21:12.754822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754302    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerDied","Data":"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"}
Sep 08 13:21:12.754822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754303    2580 scope.go:117] "RemoveContainer" containerID="34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"
Sep 08 13:21:12.754822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerDied","Data":"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"}
Sep 08 13:21:12.754822 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.754375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be32dd25-593c-46b1-a683-27582846133b","Type":"ContainerDied","Data":"2deab6357ec484935185574c7d76d4ca3eee08b6fc7490af3919e7a6a9d2d26e"}
Sep 08 13:21:12.755513 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.755492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8mnkp" event={"ID":"d011fe55-25c5-486c-850f-49d3abbd6650","Type":"ContainerStarted","Data":"471ecf3aa4ca9665a4878bbde4ca6a214ec86391535ac66407793d8771ceb558"}
Sep 08 13:21:12.763170 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.763014    2580 scope.go:117] "RemoveContainer" containerID="51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"
Sep 08 13:21:12.772069 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.772045    2580 scope.go:117] "RemoveContainer" containerID="f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"
Sep 08 13:21:12.780085 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.780061    2580 scope.go:117] "RemoveContainer" containerID="c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"
Sep 08 13:21:12.780224 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.780198    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:12.783689 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.783669    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:12.787572 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.787552    2580 scope.go:117] "RemoveContainer" containerID="92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"
Sep 08 13:21:12.795559 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.795539    2580 scope.go:117] "RemoveContainer" containerID="19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"
Sep 08 13:21:12.803753 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.803736    2580 scope.go:117] "RemoveContainer" containerID="e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"
Sep 08 13:21:12.812690 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.812670    2580 scope.go:117] "RemoveContainer" containerID="34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"
Sep 08 13:21:12.812986 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:21:12.812964    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": container with ID starting with 34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219 not found: ID does not exist" containerID="34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"
Sep 08 13:21:12.813093 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.812997    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"} err="failed to get container status \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": rpc error: code = NotFound desc = could not find container \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": container with ID starting with 34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219 not found: ID does not exist"
Sep 08 13:21:12.813093 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.813020    2580 scope.go:117] "RemoveContainer" containerID="51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"
Sep 08 13:21:12.813330 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:21:12.813306    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": container with ID starting with 51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328 not found: ID does not exist" containerID="51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"
Sep 08 13:21:12.813384 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.813339    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"} err="failed to get container status \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": rpc error: code = NotFound desc = could not find container \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": container with ID starting with 51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328 not found: ID does not exist"
Sep 08 13:21:12.813384 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.813363    2580 scope.go:117] "RemoveContainer" containerID="f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"
Sep 08 13:21:12.813674 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:21:12.813651    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": container with ID starting with f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f not found: ID does not exist" containerID="f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"
Sep 08 13:21:12.813741 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.813680    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"} err="failed to get container status \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": rpc error: code = NotFound desc = could not find container \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": container with ID starting with f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f not found: ID does not exist"
Sep 08 13:21:12.813741 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.813696    2580 scope.go:117] "RemoveContainer" containerID="c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"
Sep 08 13:21:12.814229 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:21:12.814205    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": container with ID starting with c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7 not found: ID does not exist" containerID="c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"
Sep 08 13:21:12.814289 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.814239    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"} err="failed to get container status \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": rpc error: code = NotFound desc = could not find container \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": container with ID starting with c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7 not found: ID does not exist"
Sep 08 13:21:12.814289 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.814261    2580 scope.go:117] "RemoveContainer" containerID="92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"
Sep 08 13:21:12.814617 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:21:12.814590    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": container with ID starting with 92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c not found: ID does not exist" containerID="92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"
Sep 08 13:21:12.814721 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.814625    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"} err="failed to get container status \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": rpc error: code = NotFound desc = could not find container \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": container with ID starting with 92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c not found: ID does not exist"
Sep 08 13:21:12.814721 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.814689    2580 scope.go:117] "RemoveContainer" containerID="19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"
Sep 08 13:21:12.815076 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:21:12.815049    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": container with ID starting with 19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2 not found: ID does not exist" containerID="19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"
Sep 08 13:21:12.815149 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.815105    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"} err="failed to get container status \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": rpc error: code = NotFound desc = could not find container \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": container with ID starting with 19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2 not found: ID does not exist"
Sep 08 13:21:12.815149 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.815127    2580 scope.go:117] "RemoveContainer" containerID="e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"
Sep 08 13:21:12.815860 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:21:12.815835    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": container with ID starting with e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def not found: ID does not exist" containerID="e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"
Sep 08 13:21:12.815950 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.815866    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"} err="failed to get container status \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": rpc error: code = NotFound desc = could not find container \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": container with ID starting with e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def not found: ID does not exist"
Sep 08 13:21:12.815950 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.815887    2580 scope.go:117] "RemoveContainer" containerID="34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"
Sep 08 13:21:12.816270 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.816242    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"} err="failed to get container status \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": rpc error: code = NotFound desc = could not find container \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": container with ID starting with 34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219 not found: ID does not exist"
Sep 08 13:21:12.816361 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.816272    2580 scope.go:117] "RemoveContainer" containerID="51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"
Sep 08 13:21:12.816707 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.816574    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"} err="failed to get container status \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": rpc error: code = NotFound desc = could not find container \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": container with ID starting with 51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328 not found: ID does not exist"
Sep 08 13:21:12.816809 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.816712    2580 scope.go:117] "RemoveContainer" containerID="f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"
Sep 08 13:21:12.817043 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817016    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"} err="failed to get container status \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": rpc error: code = NotFound desc = could not find container \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": container with ID starting with f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f not found: ID does not exist"
Sep 08 13:21:12.817136 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817044    2580 scope.go:117] "RemoveContainer" containerID="c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"
Sep 08 13:21:12.817306 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817287    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:12.817683 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817655    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"} err="failed to get container status \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": rpc error: code = NotFound desc = could not find container \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": container with ID starting with c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7 not found: ID does not exist"
Sep 08 13:21:12.817683 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817682    2580 scope.go:117] "RemoveContainer" containerID="92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817729    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861463ed-75d5-46a9-8aff-6de821ca7d6d" containerName="registry"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817743    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="861463ed-75d5-46a9-8aff-6de821ca7d6d" containerName="registry"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817756    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="init-config-reloader"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817765    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="init-config-reloader"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817780    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="thanos-sidecar"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817788    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="thanos-sidecar"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817801    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy-thanos"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817806    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy-thanos"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817817    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817822    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817830    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="config-reloader"
Sep 08 13:21:12.817832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817839    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="config-reloader"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817847    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy-web"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817855    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy-web"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817863    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="prometheus"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817871    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="prometheus"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817949    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="861463ed-75d5-46a9-8aff-6de821ca7d6d" containerName="registry"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817962    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy-web"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817973    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="prometheus"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817981    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="config-reloader"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817989    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="thanos-sidecar"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817997    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.818004    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be32dd25-593c-46b1-a683-27582846133b" containerName="kube-rbac-proxy-thanos"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.817999    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"} err="failed to get container status \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": rpc error: code = NotFound desc = could not find container \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": container with ID starting with 92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c not found: ID does not exist"
Sep 08 13:21:12.818386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.818171    2580 scope.go:117] "RemoveContainer" containerID="19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"
Sep 08 13:21:12.819009 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.818617    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"} err="failed to get container status \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": rpc error: code = NotFound desc = could not find container \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": container with ID starting with 19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2 not found: ID does not exist"
Sep 08 13:21:12.819009 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.818638    2580 scope.go:117] "RemoveContainer" containerID="e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"
Sep 08 13:21:12.819009 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.818944    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"} err="failed to get container status \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": rpc error: code = NotFound desc = could not find container \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": container with ID starting with e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def not found: ID does not exist"
Sep 08 13:21:12.819009 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.818980    2580 scope.go:117] "RemoveContainer" containerID="34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"
Sep 08 13:21:12.819377 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.819350    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"} err="failed to get container status \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": rpc error: code = NotFound desc = could not find container \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": container with ID starting with 34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219 not found: ID does not exist"
Sep 08 13:21:12.819377 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.819374    2580 scope.go:117] "RemoveContainer" containerID="51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"
Sep 08 13:21:12.819767 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.819741    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"} err="failed to get container status \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": rpc error: code = NotFound desc = could not find container \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": container with ID starting with 51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328 not found: ID does not exist"
Sep 08 13:21:12.819838 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.819767    2580 scope.go:117] "RemoveContainer" containerID="f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"
Sep 08 13:21:12.820082 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.820055    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"} err="failed to get container status \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": rpc error: code = NotFound desc = could not find container \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": container with ID starting with f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f not found: ID does not exist"
Sep 08 13:21:12.820082 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.820080    2580 scope.go:117] "RemoveContainer" containerID="c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"
Sep 08 13:21:12.820365 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.820338    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"} err="failed to get container status \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": rpc error: code = NotFound desc = could not find container \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": container with ID starting with c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7 not found: ID does not exist"
Sep 08 13:21:12.820480 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.820366    2580 scope.go:117] "RemoveContainer" containerID="92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"
Sep 08 13:21:12.820671 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.820645    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"} err="failed to get container status \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": rpc error: code = NotFound desc = could not find container \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": container with ID starting with 92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c not found: ID does not exist"
Sep 08 13:21:12.820724 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.820672    2580 scope.go:117] "RemoveContainer" containerID="19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"
Sep 08 13:21:12.820938 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.820911    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"} err="failed to get container status \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": rpc error: code = NotFound desc = could not find container \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": container with ID starting with 19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2 not found: ID does not exist"
Sep 08 13:21:12.820938 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.820931    2580 scope.go:117] "RemoveContainer" containerID="e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"
Sep 08 13:21:12.821192 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.821164    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"} err="failed to get container status \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": rpc error: code = NotFound desc = could not find container \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": container with ID starting with e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def not found: ID does not exist"
Sep 08 13:21:12.821255 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.821194    2580 scope.go:117] "RemoveContainer" containerID="34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"
Sep 08 13:21:12.821514 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.821487    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"} err="failed to get container status \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": rpc error: code = NotFound desc = could not find container \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": container with ID starting with 34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219 not found: ID does not exist"
Sep 08 13:21:12.821514 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.821511    2580 scope.go:117] "RemoveContainer" containerID="51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"
Sep 08 13:21:12.821806 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.821781    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"} err="failed to get container status \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": rpc error: code = NotFound desc = could not find container \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": container with ID starting with 51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328 not found: ID does not exist"
Sep 08 13:21:12.821908 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.821806    2580 scope.go:117] "RemoveContainer" containerID="f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"
Sep 08 13:21:12.822086 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.822065    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"} err="failed to get container status \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": rpc error: code = NotFound desc = could not find container \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": container with ID starting with f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f not found: ID does not exist"
Sep 08 13:21:12.822086 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.822084    2580 scope.go:117] "RemoveContainer" containerID="c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"
Sep 08 13:21:12.822333 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.822304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.822476 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.822346    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"} err="failed to get container status \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": rpc error: code = NotFound desc = could not find container \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": container with ID starting with c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7 not found: ID does not exist"
Sep 08 13:21:12.822476 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.822368    2580 scope.go:117] "RemoveContainer" containerID="92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"
Sep 08 13:21:12.822816 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.822782    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"} err="failed to get container status \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": rpc error: code = NotFound desc = could not find container \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": container with ID starting with 92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c not found: ID does not exist"
Sep 08 13:21:12.822907 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.822819    2580 scope.go:117] "RemoveContainer" containerID="19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"
Sep 08 13:21:12.823181 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.823148    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"} err="failed to get container status \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": rpc error: code = NotFound desc = could not find container \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": container with ID starting with 19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2 not found: ID does not exist"
Sep 08 13:21:12.823267 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.823183    2580 scope.go:117] "RemoveContainer" containerID="e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"
Sep 08 13:21:12.823489 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.823452    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"} err="failed to get container status \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": rpc error: code = NotFound desc = could not find container \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": container with ID starting with e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def not found: ID does not exist"
Sep 08 13:21:12.823489 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.823484    2580 scope.go:117] "RemoveContainer" containerID="34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"
Sep 08 13:21:12.823875 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.823727    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"} err="failed to get container status \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": rpc error: code = NotFound desc = could not find container \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": container with ID starting with 34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219 not found: ID does not exist"
Sep 08 13:21:12.823970 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.823901    2580 scope.go:117] "RemoveContainer" containerID="51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"
Sep 08 13:21:12.824222 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.824198    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"} err="failed to get container status \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": rpc error: code = NotFound desc = could not find container \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": container with ID starting with 51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328 not found: ID does not exist"
Sep 08 13:21:12.824299 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.824223    2580 scope.go:117] "RemoveContainer" containerID="f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"
Sep 08 13:21:12.824533 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.824507    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"} err="failed to get container status \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": rpc error: code = NotFound desc = could not find container \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": container with ID starting with f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f not found: ID does not exist"
Sep 08 13:21:12.824533 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.824533    2580 scope.go:117] "RemoveContainer" containerID="c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"
Sep 08 13:21:12.824690 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.824551    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 13:21:12.824690 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.824560    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 13:21:12.824872 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.824845    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"} err="failed to get container status \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": rpc error: code = NotFound desc = could not find container \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": container with ID starting with c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7 not found: ID does not exist"
Sep 08 13:21:12.824952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.824874    2580 scope.go:117] "RemoveContainer" containerID="92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"
Sep 08 13:21:12.825203 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825175    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"} err="failed to get container status \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": rpc error: code = NotFound desc = could not find container \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": container with ID starting with 92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c not found: ID does not exist"
Sep 08 13:21:12.825313 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825204    2580 scope.go:117] "RemoveContainer" containerID="19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"
Sep 08 13:21:12.825313 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825296    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 13:21:12.825566 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825535    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"} err="failed to get container status \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": rpc error: code = NotFound desc = could not find container \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": container with ID starting with 19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2 not found: ID does not exist"
Sep 08 13:21:12.825566 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825567    2580 scope.go:117] "RemoveContainer" containerID="e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"
Sep 08 13:21:12.825719 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825690    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 13:21:12.825801 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b8ad66i9d596i\""
Sep 08 13:21:12.826033 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825887    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 13:21:12.826033 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825936    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"} err="failed to get container status \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": rpc error: code = NotFound desc = could not find container \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": container with ID starting with e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def not found: ID does not exist"
Sep 08 13:21:12.826033 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.825959    2580 scope.go:117] "RemoveContainer" containerID="34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"
Sep 08 13:21:12.826264 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.826240    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219"} err="failed to get container status \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": rpc error: code = NotFound desc = could not find container \"34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219\": container with ID starting with 34856a2b922994bedea3362041452aee2ad45819f02672a3e05b0873784e6219 not found: ID does not exist"
Sep 08 13:21:12.826328 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.826266    2580 scope.go:117] "RemoveContainer" containerID="51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"
Sep 08 13:21:12.826570 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.826546    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328"} err="failed to get container status \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": rpc error: code = NotFound desc = could not find container \"51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328\": container with ID starting with 51a8c8735c6d7a897d3950b4f3d50f4ac1ca23e507e547d62f74ca770217d328 not found: ID does not exist"
Sep 08 13:21:12.826659 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.826572    2580 scope.go:117] "RemoveContainer" containerID="f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"
Sep 08 13:21:12.826837 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.826814    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f"} err="failed to get container status \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": rpc error: code = NotFound desc = could not find container \"f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f\": container with ID starting with f6f0e3bc7d5305a64fa26228a99d19bdb9b45bee2d6d874c94e8556dc6a9623f not found: ID does not exist"
Sep 08 13:21:12.826915 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.826838    2580 scope.go:117] "RemoveContainer" containerID="c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"
Sep 08 13:21:12.827143 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.827086    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7"} err="failed to get container status \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": rpc error: code = NotFound desc = could not find container \"c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7\": container with ID starting with c37bd4f10c6f8a8129c686d364bba3245f2b3a9b8d874ee9cb2a8aef3e4a70a7 not found: ID does not exist"
Sep 08 13:21:12.827143 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.827111    2580 scope.go:117] "RemoveContainer" containerID="92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"
Sep 08 13:21:12.827451 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.827432    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c"} err="failed to get container status \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": rpc error: code = NotFound desc = could not find container \"92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c\": container with ID starting with 92bab89b93a2caf3ff9843409153808e939915808ed91c6ac71030703fa0119c not found: ID does not exist"
Sep 08 13:21:12.827556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.827454    2580 scope.go:117] "RemoveContainer" containerID="19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"
Sep 08 13:21:12.827770 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.827748    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2"} err="failed to get container status \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": rpc error: code = NotFound desc = could not find container \"19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2\": container with ID starting with 19f727234115a7cb6022b582128c2620909a7341d3f50a01b3f3b659f2ef3ce2 not found: ID does not exist"
Sep 08 13:21:12.827864 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.827778    2580 scope.go:117] "RemoveContainer" containerID="e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"
Sep 08 13:21:12.828107 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.828088    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 13:21:12.828193 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.828149    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 13:21:12.828274 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.828248    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def"} err="failed to get container status \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": rpc error: code = NotFound desc = could not find container \"e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def\": container with ID starting with e2abe7e36f6e1cf418c860feb2a313779b99cbf625a1d6db867d707a37bc2def not found: ID does not exist"
Sep 08 13:21:12.828372 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.828295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 13:21:12.828372 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.828356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 13:21:12.828506 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.828361    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 13:21:12.828592 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.828575    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 13:21:12.828770 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.828750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 13:21:12.828974 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.828955    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qm8xt\""
Sep 08 13:21:12.831267 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.831246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 13:21:12.835190 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.835171    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 13:21:12.842424 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.842390    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:12.932081 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932081 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-web-config\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932081 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932299 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932299 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932299 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932299 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjzh9\" (UniqueName: \"kubernetes.io/projected/78ad7b9c-2f97-431b-b270-554d8e1e2913-kube-api-access-rjzh9\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932299 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78ad7b9c-2f97-431b-b270-554d8e1e2913-config-out\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-config\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78ad7b9c-2f97-431b-b270-554d8e1e2913-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932526    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932854 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932854 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932854 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932854 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.932854 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:12.932729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.033674 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.033857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.033857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.033857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.033857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.033857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.033857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.033857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-web-config\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.033993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjzh9\" (UniqueName: \"kubernetes.io/projected/78ad7b9c-2f97-431b-b270-554d8e1e2913-kube-api-access-rjzh9\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78ad7b9c-2f97-431b-b270-554d8e1e2913-config-out\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-config\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78ad7b9c-2f97-431b-b270-554d8e1e2913-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034199 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034783 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034783 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034891 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.034974 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.034948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.036210 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.035851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.037760 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.037646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.037760 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.037732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.038233 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.038209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.038517 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.038493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78ad7b9c-2f97-431b-b270-554d8e1e2913-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.038942 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.038924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.039963 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.039497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.039963 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.039560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.039963 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.039917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.040375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.040350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-web-config\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.040489 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.040354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78ad7b9c-2f97-431b-b270-554d8e1e2913-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.040686 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.040662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-config\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.040989 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.040947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.041536 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.041515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78ad7b9c-2f97-431b-b270-554d8e1e2913-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.043006 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.042984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78ad7b9c-2f97-431b-b270-554d8e1e2913-config-out\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.046310 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.046287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjzh9\" (UniqueName: \"kubernetes.io/projected/78ad7b9c-2f97-431b-b270-554d8e1e2913-kube-api-access-rjzh9\") pod \"prometheus-k8s-0\" (UID: \"78ad7b9c-2f97-431b-b270-554d8e1e2913\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.136259 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.136225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:13.430233 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.430072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:13.750720 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:13.750686    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be32dd25-593c-46b1-a683-27582846133b" path="/var/lib/kubelet/pods/be32dd25-593c-46b1-a683-27582846133b/volumes"
Sep 08 13:21:13.833266 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:21:13.833234    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ad7b9c_2f97_431b_b270_554d8e1e2913.slice/crio-49cae6d556e044541f6a427c037491a1d49a7d9f27e216011695deb45701b63f WatchSource:0}: Error finding container 49cae6d556e044541f6a427c037491a1d49a7d9f27e216011695deb45701b63f: Status 404 returned error can't find the container with id 49cae6d556e044541f6a427c037491a1d49a7d9f27e216011695deb45701b63f
Sep 08 13:21:14.750573 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:14.750531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:14.766275 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:14.766242    2580 generic.go:356] "Generic (PLEG): container finished" podID="78ad7b9c-2f97-431b-b270-554d8e1e2913" containerID="0393942eab0d41b698d7b4c73023e1740848d9d8a9546e106e925362f4609d52" exitCode=0
Sep 08 13:21:14.766450 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:14.766324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78ad7b9c-2f97-431b-b270-554d8e1e2913","Type":"ContainerDied","Data":"0393942eab0d41b698d7b4c73023e1740848d9d8a9546e106e925362f4609d52"}
Sep 08 13:21:14.766506 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:14.766468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78ad7b9c-2f97-431b-b270-554d8e1e2913","Type":"ContainerStarted","Data":"49cae6d556e044541f6a427c037491a1d49a7d9f27e216011695deb45701b63f"}
Sep 08 13:21:14.768286 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:14.768254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8mnkp" event={"ID":"d011fe55-25c5-486c-850f-49d3abbd6650","Type":"ContainerStarted","Data":"47b14bfaaa4012a8004712a0ec9529c9b4c359de9e4f41f778c33cc3f3160b46"}
Sep 08 13:21:14.768376 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:14.768301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8mnkp" event={"ID":"d011fe55-25c5-486c-850f-49d3abbd6650","Type":"ContainerStarted","Data":"68989ca95c73c27a91bbf3ca75ef7f06f4921fd2dd7faf07880e248d9d7361d0"}
Sep 08 13:21:14.768376 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:14.768335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:21:14.816077 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:14.816052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:14.821656 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:14.821612    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8mnkp" podStartSLOduration=252.047384168 podStartE2EDuration="4m13.82159476s" totalImagesPullingTime="1.774210592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:14.820526533 +0000 UTC m=+287.520457178" watchObservedRunningTime="2026-09-08 13:21:14.82159476 +0000 UTC m=+287.521525599"
Sep 08 13:21:15.776429 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:15.776373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78ad7b9c-2f97-431b-b270-554d8e1e2913","Type":"ContainerStarted","Data":"51709034ec57ea1b7a1debdbb9f911ad745ef4e92b5e5307fc7b67a2a055ef6c"}
Sep 08 13:21:15.776770 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:15.776471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78ad7b9c-2f97-431b-b270-554d8e1e2913","Type":"ContainerStarted","Data":"2527cae2a88f184139e804632f444e0b4f5c79890b478c8bf7d12d1f8b667f1a"}
Sep 08 13:21:15.776770 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:15.776512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78ad7b9c-2f97-431b-b270-554d8e1e2913","Type":"ContainerStarted","Data":"4a0ada4c672100fdf0e2fcfb359f90aa398d3e93cc2aa6165a7e8b41a75b193c"}
Sep 08 13:21:15.776770 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:15.776540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78ad7b9c-2f97-431b-b270-554d8e1e2913","Type":"ContainerStarted","Data":"63b5a61ee2dbbd450ab44ca3e93f173ac0c8ad8a2042cbe0af0da6b8012a2473"}
Sep 08 13:21:15.776770 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:15.776563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78ad7b9c-2f97-431b-b270-554d8e1e2913","Type":"ContainerStarted","Data":"0a98f26e7920829fd3df1c0d97c9e4ee09e72e6bc08f2c4169880b4bbd290eae"}
Sep 08 13:21:15.776770 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:15.776590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78ad7b9c-2f97-431b-b270-554d8e1e2913","Type":"ContainerStarted","Data":"0814e1f03eebab5109d87477616d1e3e9c8ae8b130883ef5e32091cacdcb243f"}
Sep 08 13:21:15.810157 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:15.810103    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.788029648 podStartE2EDuration="3.810083342s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.053694ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:15.808756861 +0000 UTC m=+288.508687530" watchObservedRunningTime="2026-09-08 13:21:15.810083342 +0000 UTC m=+288.510014002"
Sep 08 13:21:18.137386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:18.137346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:21.383699 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:21.383658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:24.779119 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:24.779087    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8mnkp"
Sep 08 13:21:27.677603 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:27.677582    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 13:21:28.150805 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:28.150772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:33.930420 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:33.930377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:34.004597 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:21:34.004551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:08.055179 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:08.055143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:13.136460 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:13.136414    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:22:13.151999 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:13.151967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:22:14.014568 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:14.014540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:22:21.857479 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:21.857382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:21.992022 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:21.991978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:25.477772 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:25.477736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.158224 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:37.158181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.221104 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:37.221074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.660946 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:37.660909    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.764769 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:37.764738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.989038 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:37.988947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:51.097782 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:22:51.097747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:50.162224 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:23:50.162142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:50.260631 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:23:50.260598    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:50.369132 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:23:50.369098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:57.700616 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:23:57.700577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:57.768755 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:23:57.768718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:18.574770 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:24:18.574737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:18.684218 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:24:18.684177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:24.626552 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:24:24.626512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:24.961872 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:24:24.961788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:25.131367 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:24:25.131320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:25.496930 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:24:25.496894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:28.590629 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:24:28.590596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:28.690615 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:24:28.690584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:02.681872 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:02.681835    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:02.793287 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:02.793252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:04.403689 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:04.403652    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:04.465619 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:04.465586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:16.798052 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:16.798017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:16.881712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:16.881679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:16.985008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:16.984973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:17.134543 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:17.134507    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:17.505507 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:17.505428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:17.568886 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:17.568855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:17.630078 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:17.630045    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:27.782938 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:27.782866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:29.579180 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:29.579141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:29.637431 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:29.637388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:45.775512 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:45.775470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:45.847589 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:45.847544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:45.909819 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:45.909780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:46.007416 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:46.007375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:46.109016 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:46.108987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:46.309468 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:46.309437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:46.410468 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:46.410359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:46.510829 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:25:46.510795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:05.265191 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:05.265100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:07.776607 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:07.776572    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:16.191849 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:16.191807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:25.980352 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:25.980317    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-nt9s7"]
Sep 08 13:27:25.983629 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:25.983610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nt9s7"
Sep 08 13:27:25.985775 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:25.985753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 08 13:27:25.985910 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:25.985811    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 08 13:27:25.985910 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:25.985844    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 08 13:27:25.985910 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:25.985873    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 08 13:27:25.986465 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:25.986446    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-j4d29\""
Sep 08 13:27:25.991928 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:25.991906    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nt9s7"]
Sep 08 13:27:26.046281 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:26.046243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3d1256-34af-4575-b90a-1b0bd45e3707-model-cache\") pod \"s3-init-nt9s7\" (UID: \"fe3d1256-34af-4575-b90a-1b0bd45e3707\") " pod="kserve/s3-init-nt9s7"
Sep 08 13:27:26.046478 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:26.046310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5zl\" (UniqueName: \"kubernetes.io/projected/fe3d1256-34af-4575-b90a-1b0bd45e3707-kube-api-access-nj5zl\") pod \"s3-init-nt9s7\" (UID: \"fe3d1256-34af-4575-b90a-1b0bd45e3707\") " pod="kserve/s3-init-nt9s7"
Sep 08 13:27:26.147547 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:26.147504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3d1256-34af-4575-b90a-1b0bd45e3707-model-cache\") pod \"s3-init-nt9s7\" (UID: \"fe3d1256-34af-4575-b90a-1b0bd45e3707\") " pod="kserve/s3-init-nt9s7"
Sep 08 13:27:26.147673 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:26.147593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5zl\" (UniqueName: \"kubernetes.io/projected/fe3d1256-34af-4575-b90a-1b0bd45e3707-kube-api-access-nj5zl\") pod \"s3-init-nt9s7\" (UID: \"fe3d1256-34af-4575-b90a-1b0bd45e3707\") " pod="kserve/s3-init-nt9s7"
Sep 08 13:27:26.147925 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:26.147903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3d1256-34af-4575-b90a-1b0bd45e3707-model-cache\") pod \"s3-init-nt9s7\" (UID: \"fe3d1256-34af-4575-b90a-1b0bd45e3707\") " pod="kserve/s3-init-nt9s7"
Sep 08 13:27:26.155606 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:26.155581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5zl\" (UniqueName: \"kubernetes.io/projected/fe3d1256-34af-4575-b90a-1b0bd45e3707-kube-api-access-nj5zl\") pod \"s3-init-nt9s7\" (UID: \"fe3d1256-34af-4575-b90a-1b0bd45e3707\") " pod="kserve/s3-init-nt9s7"
Sep 08 13:27:26.308151 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:26.308064    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nt9s7"
Sep 08 13:27:26.577105 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:26.577068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nt9s7"]
Sep 08 13:27:26.579364 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:27:26.579336    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe3d1256_34af_4575_b90a_1b0bd45e3707.slice/crio-fdb4d58c7e712a80081c6db785bc5622ba34f31877254be347747c03b9c61b22 WatchSource:0}: Error finding container fdb4d58c7e712a80081c6db785bc5622ba34f31877254be347747c03b9c61b22: Status 404 returned error can't find the container with id fdb4d58c7e712a80081c6db785bc5622ba34f31877254be347747c03b9c61b22
Sep 08 13:27:26.581175 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:26.581157    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:27:27.215023 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:27.214989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nt9s7" event={"ID":"fe3d1256-34af-4575-b90a-1b0bd45e3707","Type":"ContainerStarted","Data":"fdb4d58c7e712a80081c6db785bc5622ba34f31877254be347747c03b9c61b22"}
Sep 08 13:27:30.096814 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:30.096791    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 08 13:27:30.234322 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:30.234279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nt9s7" event={"ID":"fe3d1256-34af-4575-b90a-1b0bd45e3707","Type":"ContainerStarted","Data":"f33a46c8e5e19f877aec57c7244d3bbb60013b3bc8be5292c899333628b59c8a"}
Sep 08 13:27:33.248523 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:33.248491    2580 generic.go:356] "Generic (PLEG): container finished" podID="fe3d1256-34af-4575-b90a-1b0bd45e3707" containerID="f33a46c8e5e19f877aec57c7244d3bbb60013b3bc8be5292c899333628b59c8a" exitCode=0
Sep 08 13:27:33.248900 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:33.248566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nt9s7" event={"ID":"fe3d1256-34af-4575-b90a-1b0bd45e3707","Type":"ContainerDied","Data":"f33a46c8e5e19f877aec57c7244d3bbb60013b3bc8be5292c899333628b59c8a"}
Sep 08 13:27:37.654792 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:37.654770    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 08 13:27:38.275640 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:38.275604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nt9s7" event={"ID":"fe3d1256-34af-4575-b90a-1b0bd45e3707","Type":"ContainerStarted","Data":"9f23faddcd845e485679d4d14a8def0281ca6032786129e158139c44befed3e5"}
Sep 08 13:27:38.293510 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:38.293459    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-nt9s7" podStartSLOduration=3.300619547 podStartE2EDuration="13.293445118s" totalImagesPullingTime="7.909870728s" totalInitContainerRuntime="2.082954843s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:27:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:38.291721571 +0000 UTC m=+670.991652204" watchObservedRunningTime="2026-09-08 13:27:38.293445118 +0000 UTC m=+670.993375755"
Sep 08 13:27:44.482032 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:27:44.481993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:43.528566 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:43.528533    2580 generic.go:356] "Generic (PLEG): container finished" podID="fe3d1256-34af-4575-b90a-1b0bd45e3707" containerID="9f23faddcd845e485679d4d14a8def0281ca6032786129e158139c44befed3e5" exitCode=0
Sep 08 13:28:43.529058 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:43.528610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nt9s7" event={"ID":"fe3d1256-34af-4575-b90a-1b0bd45e3707","Type":"ContainerDied","Data":"9f23faddcd845e485679d4d14a8def0281ca6032786129e158139c44befed3e5"}
Sep 08 13:28:44.653010 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:44.652984    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nt9s7"
Sep 08 13:28:44.821481 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:44.821354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3d1256-34af-4575-b90a-1b0bd45e3707-model-cache\") pod \"fe3d1256-34af-4575-b90a-1b0bd45e3707\" (UID: \"fe3d1256-34af-4575-b90a-1b0bd45e3707\") "
Sep 08 13:28:44.821636 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:44.821512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj5zl\" (UniqueName: \"kubernetes.io/projected/fe3d1256-34af-4575-b90a-1b0bd45e3707-kube-api-access-nj5zl\") pod \"fe3d1256-34af-4575-b90a-1b0bd45e3707\" (UID: \"fe3d1256-34af-4575-b90a-1b0bd45e3707\") "
Sep 08 13:28:44.823767 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:44.823734    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3d1256-34af-4575-b90a-1b0bd45e3707-kube-api-access-nj5zl" (OuterVolumeSpecName: "kube-api-access-nj5zl") pod "fe3d1256-34af-4575-b90a-1b0bd45e3707" (UID: "fe3d1256-34af-4575-b90a-1b0bd45e3707"). InnerVolumeSpecName "kube-api-access-nj5zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:28:44.840799 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:44.840755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3d1256-34af-4575-b90a-1b0bd45e3707-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe3d1256-34af-4575-b90a-1b0bd45e3707" (UID: "fe3d1256-34af-4575-b90a-1b0bd45e3707"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:28:44.922904 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:44.922875    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5zl\" (UniqueName: \"kubernetes.io/projected/fe3d1256-34af-4575-b90a-1b0bd45e3707-kube-api-access-nj5zl\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:28:44.922904 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:44.922903    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3d1256-34af-4575-b90a-1b0bd45e3707-model-cache\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:28:45.537278 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:45.537249    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nt9s7"
Sep 08 13:28:45.537460 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:45.537249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nt9s7" event={"ID":"fe3d1256-34af-4575-b90a-1b0bd45e3707","Type":"ContainerDied","Data":"fdb4d58c7e712a80081c6db785bc5622ba34f31877254be347747c03b9c61b22"}
Sep 08 13:28:45.537460 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:45.537367    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdb4d58c7e712a80081c6db785bc5622ba34f31877254be347747c03b9c61b22"
Sep 08 13:28:45.910944 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:45.910913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:46.011754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:46.011711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:57.673555 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.673513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:57.781582 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.781549    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"]
Sep 08 13:28:57.781881 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.781870    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe3d1256-34af-4575-b90a-1b0bd45e3707" containerName="s3-init"
Sep 08 13:28:57.781923 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.781884    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe3d1256-34af-4575-b90a-1b0bd45e3707" containerName="s3-init"
Sep 08 13:28:57.781923 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.781892    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe3d1256-34af-4575-b90a-1b0bd45e3707" containerName="download-hf-model"
Sep 08 13:28:57.781923 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.781898    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe3d1256-34af-4575-b90a-1b0bd45e3707" containerName="download-hf-model"
Sep 08 13:28:57.782018 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.781958    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe3d1256-34af-4575-b90a-1b0bd45e3707" containerName="s3-init"
Sep 08 13:28:57.784009 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.783994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:57.786541 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.786521    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f794f-predictor-serving-cert\""
Sep 08 13:28:57.787266 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.787244    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:28:57.787454 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.787305    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f794f-kube-rbac-proxy-sar-config\""
Sep 08 13:28:57.787454 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.787330    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:28:57.787454 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.787434    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pw8wg\""
Sep 08 13:28:57.796189 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.796163    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"]
Sep 08 13:28:57.819527 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.819492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4543f3-e757-4b16-978f-862868529523-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:57.819679 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.819535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7czq\" (UniqueName: \"kubernetes.io/projected/ca4543f3-e757-4b16-978f-862868529523-kube-api-access-d7czq\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:57.819679 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.819556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4543f3-e757-4b16-978f-862868529523-proxy-tls\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:57.920286 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.920246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4543f3-e757-4b16-978f-862868529523-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:57.920550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.920312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7czq\" (UniqueName: \"kubernetes.io/projected/ca4543f3-e757-4b16-978f-862868529523-kube-api-access-d7czq\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:57.920550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.920346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4543f3-e757-4b16-978f-862868529523-proxy-tls\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:57.920550 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:28:57.920495    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f794f-predictor-serving-cert: secret "success-200-isvc-f794f-predictor-serving-cert" not found
Sep 08 13:28:57.920721 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:28:57.920555    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca4543f3-e757-4b16-978f-862868529523-proxy-tls podName:ca4543f3-e757-4b16-978f-862868529523 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:58.420536695 +0000 UTC m=+751.120467306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ca4543f3-e757-4b16-978f-862868529523-proxy-tls") pod "success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" (UID: "ca4543f3-e757-4b16-978f-862868529523") : secret "success-200-isvc-f794f-predictor-serving-cert" not found
Sep 08 13:28:57.921127 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.921088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4543f3-e757-4b16-978f-862868529523-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:57.931663 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.931606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7czq\" (UniqueName: \"kubernetes.io/projected/ca4543f3-e757-4b16-978f-862868529523-kube-api-access-d7czq\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:57.971938 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:57.971894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:58.009949 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.009913    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"]
Sep 08 13:28:58.013662 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.013645    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.018054 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.018027    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Sep 08 13:28:58.018178 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.018038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:28:58.018493 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.018474    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Sep 08 13:28:58.018586 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.018476    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:28:58.020882 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.020858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vkg\" (UniqueName: \"kubernetes.io/projected/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-api-access-q5vkg\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.020988 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.020939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9deb1efd-bcdf-40e2-9410-0a86324765a1-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.021052 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.020994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.021052 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.021026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.021148 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.021102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9deb1efd-bcdf-40e2-9410-0a86324765a1-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.030112 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.030087    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"]
Sep 08 13:28:58.092028 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.091988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:58.121604 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.121566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9deb1efd-bcdf-40e2-9410-0a86324765a1-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.121785 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.121620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.121785 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.121653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.121785 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.121695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9deb1efd-bcdf-40e2-9410-0a86324765a1-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.121785 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.121768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vkg\" (UniqueName: \"kubernetes.io/projected/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-api-access-q5vkg\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.121984 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:28:58.121895    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 08 13:28:58.121984 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:28:58.121978    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9deb1efd-bcdf-40e2-9410-0a86324765a1-proxy-tls podName:9deb1efd-bcdf-40e2-9410-0a86324765a1 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:58.621955255 +0000 UTC m=+751.321885870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9deb1efd-bcdf-40e2-9410-0a86324765a1-proxy-tls") pod "isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" (UID: "9deb1efd-bcdf-40e2-9410-0a86324765a1") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 08 13:28:58.122096 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.122047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9deb1efd-bcdf-40e2-9410-0a86324765a1-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.122362 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.122344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.122470 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.122447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.140258 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.140232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vkg\" (UniqueName: \"kubernetes.io/projected/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-api-access-q5vkg\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.193438 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.193336    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"]
Sep 08 13:28:58.196240 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.196215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.198787 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.198766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 08 13:28:58.199736 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.199718    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 08 13:28:58.213980 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.213959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"]
Sep 08 13:28:58.222873 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.222849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.222969 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.222929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83493665-53a4-4bed-8f89-0f2d1940bc18-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.223008 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.222963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.223043 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.223023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkx7s\" (UniqueName: \"kubernetes.io/projected/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-api-access-jkx7s\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.223101 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.223086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83493665-53a4-4bed-8f89-0f2d1940bc18-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.262164 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.262125    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:58.323599 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.323567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.323761 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.323617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkx7s\" (UniqueName: \"kubernetes.io/projected/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-api-access-jkx7s\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.323761 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.323689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83493665-53a4-4bed-8f89-0f2d1940bc18-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.323761 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.323736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.323898 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.323799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83493665-53a4-4bed-8f89-0f2d1940bc18-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.323898 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:28:58.323839    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 08 13:28:58.323998 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:28:58.323915    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83493665-53a4-4bed-8f89-0f2d1940bc18-proxy-tls podName:83493665-53a4-4bed-8f89-0f2d1940bc18 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:58.823894771 +0000 UTC m=+751.523825394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/83493665-53a4-4bed-8f89-0f2d1940bc18-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-d85h2" (UID: "83493665-53a4-4bed-8f89-0f2d1940bc18") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 08 13:28:58.324118 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.324096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83493665-53a4-4bed-8f89-0f2d1940bc18-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.324365 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.324343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.324520 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.324501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.339918 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.339893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkx7s\" (UniqueName: \"kubernetes.io/projected/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-api-access-jkx7s\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.382240 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.382205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:58.424514 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.424480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4543f3-e757-4b16-978f-862868529523-proxy-tls\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:58.426794 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.426774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4543f3-e757-4b16-978f-862868529523-proxy-tls\") pod \"success-200-isvc-f794f-predictor-75b57f85bc-4s5nf\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") " pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:58.460708 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.460630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:58.625784 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.625749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9deb1efd-bcdf-40e2-9410-0a86324765a1-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.628360 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.628334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9deb1efd-bcdf-40e2-9410-0a86324765a1-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.694344 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.694302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:28:58.827099 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.827070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83493665-53a4-4bed-8f89-0f2d1940bc18-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.829464 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.829436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83493665-53a4-4bed-8f89-0f2d1940bc18-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-d85h2\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:58.881102 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.881068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:58.927001 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.926971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:28:58.971296 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.971265    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"]
Sep 08 13:28:58.974923 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:58.974853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:28:58.975280 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:28:58.975217    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4543f3_e757_4b16_978f_862868529523.slice/crio-dbe2944031e97b492ea7d84345fa8a3326b70bb5acf32b652979498b379068f2 WatchSource:0}: Error finding container dbe2944031e97b492ea7d84345fa8a3326b70bb5acf32b652979498b379068f2: Status 404 returned error can't find the container with id dbe2944031e97b492ea7d84345fa8a3326b70bb5acf32b652979498b379068f2
Sep 08 13:28:59.107556 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:59.107515    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:28:59.202379 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:59.202347    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"]
Sep 08 13:28:59.205320 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:28:59.205282    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9deb1efd_bcdf_40e2_9410_0a86324765a1.slice/crio-52c406bc14c092f5b414434311420e4d199430f9ddfb910aa6d27a76acabb26b WatchSource:0}: Error finding container 52c406bc14c092f5b414434311420e4d199430f9ddfb910aa6d27a76acabb26b: Status 404 returned error can't find the container with id 52c406bc14c092f5b414434311420e4d199430f9ddfb910aa6d27a76acabb26b
Sep 08 13:28:59.379430 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:59.379407    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"]
Sep 08 13:28:59.382904 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:28:59.382876    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83493665_53a4_4bed_8f89_0f2d1940bc18.slice/crio-28baf33fc9927e97d9085269473282fd7c30cdd162e55cc91ee18609c740b3f3 WatchSource:0}: Error finding container 28baf33fc9927e97d9085269473282fd7c30cdd162e55cc91ee18609c740b3f3: Status 404 returned error can't find the container with id 28baf33fc9927e97d9085269473282fd7c30cdd162e55cc91ee18609c740b3f3
Sep 08 13:28:59.596000 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:59.595962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" event={"ID":"83493665-53a4-4bed-8f89-0f2d1940bc18","Type":"ContainerStarted","Data":"d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797"}
Sep 08 13:28:59.596196 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:59.596045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" event={"ID":"83493665-53a4-4bed-8f89-0f2d1940bc18","Type":"ContainerStarted","Data":"28baf33fc9927e97d9085269473282fd7c30cdd162e55cc91ee18609c740b3f3"}
Sep 08 13:28:59.597754 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:59.597725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" event={"ID":"9deb1efd-bcdf-40e2-9410-0a86324765a1","Type":"ContainerStarted","Data":"bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121"}
Sep 08 13:28:59.597885 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:59.597790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" event={"ID":"9deb1efd-bcdf-40e2-9410-0a86324765a1","Type":"ContainerStarted","Data":"52c406bc14c092f5b414434311420e4d199430f9ddfb910aa6d27a76acabb26b"}
Sep 08 13:28:59.599099 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:28:59.599068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" event={"ID":"ca4543f3-e757-4b16-978f-862868529523","Type":"ContainerStarted","Data":"dbe2944031e97b492ea7d84345fa8a3326b70bb5acf32b652979498b379068f2"}
Sep 08 13:29:04.629027 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:04.628981    2580 generic.go:356] "Generic (PLEG): container finished" podID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerID="d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797" exitCode=0
Sep 08 13:29:04.629509 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:04.629062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" event={"ID":"83493665-53a4-4bed-8f89-0f2d1940bc18","Type":"ContainerDied","Data":"d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797"}
Sep 08 13:29:04.631162 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:04.631131    2580 generic.go:356] "Generic (PLEG): container finished" podID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerID="bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121" exitCode=0
Sep 08 13:29:04.631291 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:04.631186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" event={"ID":"9deb1efd-bcdf-40e2-9410-0a86324765a1","Type":"ContainerDied","Data":"bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121"}
Sep 08 13:29:11.669348 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:11.669215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" event={"ID":"ca4543f3-e757-4b16-978f-862868529523","Type":"ContainerStarted","Data":"701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430"}
Sep 08 13:29:20.722812 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.722775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" event={"ID":"9deb1efd-bcdf-40e2-9410-0a86324765a1","Type":"ContainerStarted","Data":"aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac"}
Sep 08 13:29:20.723301 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.722855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" event={"ID":"9deb1efd-bcdf-40e2-9410-0a86324765a1","Type":"ContainerStarted","Data":"a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d"}
Sep 08 13:29:20.723301 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.723231    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:29:20.723439 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.723368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:29:20.724701 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.724671    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:20.725339 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.725311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" event={"ID":"ca4543f3-e757-4b16-978f-862868529523","Type":"ContainerStarted","Data":"9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b"}
Sep 08 13:29:20.725479 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.725378    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:29:20.725479 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.725431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:29:20.726326 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.726300    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:20.745342 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.745298    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podStartSLOduration=4.651855948 podStartE2EDuration="23.745283987s" totalImagesPullingTime="15.105750777s" totalInitContainerRuntime="3.987677262s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:20.742971194 +0000 UTC m=+773.442901829" watchObservedRunningTime="2026-09-08 13:29:20.745283987 +0000 UTC m=+773.445214621"
Sep 08 13:29:20.760817 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:20.760761    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" podStartSLOduration=3.10911105 podStartE2EDuration="23.760744302s" totalImagesPullingTime="20.651633252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:20.759382053 +0000 UTC m=+773.459312714" watchObservedRunningTime="2026-09-08 13:29:20.760744302 +0000 UTC m=+773.460674938"
Sep 08 13:29:21.733619 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:21.733579    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:21.734074 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:21.733654    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:26.739316 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:26.739011    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:29:26.739936 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:26.739839    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:26.740774 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:26.740750    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:29:26.741343 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:26.741309    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:28.771048 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:28.770958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" event={"ID":"83493665-53a4-4bed-8f89-0f2d1940bc18","Type":"ContainerStarted","Data":"1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27"}
Sep 08 13:29:28.771048 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:28.771023    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" event={"ID":"83493665-53a4-4bed-8f89-0f2d1940bc18","Type":"ContainerStarted","Data":"1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650"}
Sep 08 13:29:28.771485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:28.771288    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:29:28.771485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:28.771455    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:29:28.772613 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:28.772584    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:29:28.790628 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:28.790568    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podStartSLOduration=2.925316493 podStartE2EDuration="30.7905537s" totalImagesPullingTime="23.727221475s" totalInitContainerRuntime="4.138015732s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:28.789072378 +0000 UTC m=+781.489003011" watchObservedRunningTime="2026-09-08 13:29:28.7905537 +0000 UTC m=+781.490484332"
Sep 08 13:29:29.774855 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:29.774819    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:29:34.779387 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:34.779351    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:29:34.779909 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:34.779882    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:29:36.739864 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:36.739824    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:36.742120 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:36.742088    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:44.438771 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:44.438731    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:44.780330 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:44.780248    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:29:46.739904 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:46.739869    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:46.742143 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:46.742119    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:54.780667 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:54.780592    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:29:56.740037 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:56.740000    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:56.741259 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:29:56.741226    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:30:04.779841 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:04.779795    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:30:06.740493 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:06.740463    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:30:06.741297 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:06.741276    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:30:14.779876 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:14.779836    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:30:16.741377 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:16.741338    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:30:17.801210 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:17.801170    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:17.947867 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:17.947834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:24.780902 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:24.780852    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:30:26.741265 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:26.741227    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:30:31.968601 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:31.968551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:32.076784 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.076753    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"]
Sep 08 13:30:32.077030 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.077001    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container" containerID="cri-o://701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430" gracePeriod=30
Sep 08 13:30:32.077090 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.077023    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kube-rbac-proxy" containerID="cri-o://9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b" gracePeriod=30
Sep 08 13:30:32.115475 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.115439    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:32.163281 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.163246    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"]
Sep 08 13:30:32.167048 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.167022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.169336 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.169260    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-87be3-kube-rbac-proxy-sar-config\""
Sep 08 13:30:32.169506 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.169359    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-87be3-predictor-serving-cert\""
Sep 08 13:30:32.175696 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.175624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"]
Sep 08 13:30:32.257985 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.257890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2s46\" (UniqueName: \"kubernetes.io/projected/7cda03f3-9df3-4741-8799-8874f44370b4-kube-api-access-s2s46\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.257985 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.257960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.257985 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.257982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cda03f3-9df3-4741-8799-8874f44370b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.358914 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.358877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.359120 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.358921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cda03f3-9df3-4741-8799-8874f44370b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.359120 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.359014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2s46\" (UniqueName: \"kubernetes.io/projected/7cda03f3-9df3-4741-8799-8874f44370b4-kube-api-access-s2s46\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.359120 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:30:32.359019    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-87be3-predictor-serving-cert: secret "success-200-isvc-87be3-predictor-serving-cert" not found
Sep 08 13:30:32.359120 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:30:32.359093    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls podName:7cda03f3-9df3-4741-8799-8874f44370b4 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:32.859073053 +0000 UTC m=+845.559003665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls") pod "success-200-isvc-87be3-predictor-845b844549-cpcg2" (UID: "7cda03f3-9df3-4741-8799-8874f44370b4") : secret "success-200-isvc-87be3-predictor-serving-cert" not found
Sep 08 13:30:32.359815 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.359794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cda03f3-9df3-4741-8799-8874f44370b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.369054 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.369027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2s46\" (UniqueName: \"kubernetes.io/projected/7cda03f3-9df3-4741-8799-8874f44370b4-kube-api-access-s2s46\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.862423 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:32.862340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:32.862610 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:30:32.862501    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-87be3-predictor-serving-cert: secret "success-200-isvc-87be3-predictor-serving-cert" not found
Sep 08 13:30:32.862610 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:30:32.862567    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls podName:7cda03f3-9df3-4741-8799-8874f44370b4 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:33.862551837 +0000 UTC m=+846.562482450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls") pod "success-200-isvc-87be3-predictor-845b844549-cpcg2" (UID: "7cda03f3-9df3-4741-8799-8874f44370b4") : secret "success-200-isvc-87be3-predictor-serving-cert" not found
Sep 08 13:30:33.041444 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:33.041382    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca4543f3-e757-4b16-978f-862868529523" containerID="9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b" exitCode=2
Sep 08 13:30:33.041817 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:33.041458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" event={"ID":"ca4543f3-e757-4b16-978f-862868529523","Type":"ContainerDied","Data":"9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b"}
Sep 08 13:30:33.061026 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:33.060995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:33.870166 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:33.870123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:33.872505 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:33.872475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls\") pod \"success-200-isvc-87be3-predictor-845b844549-cpcg2\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") " pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:33.980108 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:33.980067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:34.251007 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:34.250930    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"]
Sep 08 13:30:34.254444 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:30:34.254417    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cda03f3_9df3_4741_8799_8874f44370b4.slice/crio-b2aee55d843a809dcbeaa8d6b743eb3789921d9635bcc054b2e2c2e8cbfaf46f WatchSource:0}: Error finding container b2aee55d843a809dcbeaa8d6b743eb3789921d9635bcc054b2e2c2e8cbfaf46f: Status 404 returned error can't find the container with id b2aee55d843a809dcbeaa8d6b743eb3789921d9635bcc054b2e2c2e8cbfaf46f
Sep 08 13:30:34.780448 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:34.780385    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:30:35.053418 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.053310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" event={"ID":"7cda03f3-9df3-4741-8799-8874f44370b4","Type":"ContainerStarted","Data":"0bf11a7051792defaa5a5e0152f5782f9462d13b6b4122c6f292e9f3f48a3a9d"}
Sep 08 13:30:35.053418 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.053369    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:35.053418 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.053390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" event={"ID":"7cda03f3-9df3-4741-8799-8874f44370b4","Type":"ContainerStarted","Data":"6858f1ac9898921f877845d46990e84169608410fe5d02dcbc53d04659e72610"}
Sep 08 13:30:35.053679 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.053426    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:35.053679 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.053450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" event={"ID":"7cda03f3-9df3-4741-8799-8874f44370b4","Type":"ContainerStarted","Data":"b2aee55d843a809dcbeaa8d6b743eb3789921d9635bcc054b2e2c2e8cbfaf46f"}
Sep 08 13:30:35.054708 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.054681    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:30:35.073010 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.072962    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" podStartSLOduration=3.072946567 podStartE2EDuration="3.072946567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:35.071268322 +0000 UTC m=+847.771198957" watchObservedRunningTime="2026-09-08 13:30:35.072946567 +0000 UTC m=+847.772877200"
Sep 08 13:30:35.381330 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.381293    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:35.420369 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.420342    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:30:35.483540 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.483508    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7czq\" (UniqueName: \"kubernetes.io/projected/ca4543f3-e757-4b16-978f-862868529523-kube-api-access-d7czq\") pod \"ca4543f3-e757-4b16-978f-862868529523\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") "
Sep 08 13:30:35.483734 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.483575    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4543f3-e757-4b16-978f-862868529523-proxy-tls\") pod \"ca4543f3-e757-4b16-978f-862868529523\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") "
Sep 08 13:30:35.483734 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.483608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4543f3-e757-4b16-978f-862868529523-kube-rbac-proxy-sar-config\") pod \"ca4543f3-e757-4b16-978f-862868529523\" (UID: \"ca4543f3-e757-4b16-978f-862868529523\") "
Sep 08 13:30:35.483961 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.483939    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4543f3-e757-4b16-978f-862868529523-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca4543f3-e757-4b16-978f-862868529523" (UID: "ca4543f3-e757-4b16-978f-862868529523"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:30:35.485773 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.485748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4543f3-e757-4b16-978f-862868529523-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca4543f3-e757-4b16-978f-862868529523" (UID: "ca4543f3-e757-4b16-978f-862868529523"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:30:35.485881 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.485864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4543f3-e757-4b16-978f-862868529523-kube-api-access-d7czq" (OuterVolumeSpecName: "kube-api-access-d7czq") pod "ca4543f3-e757-4b16-978f-862868529523" (UID: "ca4543f3-e757-4b16-978f-862868529523"). InnerVolumeSpecName "kube-api-access-d7czq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:30:35.584442 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.584376    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4543f3-e757-4b16-978f-862868529523-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:30:35.584442 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.584435    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4543f3-e757-4b16-978f-862868529523-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:30:35.584442 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:35.584445    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7czq\" (UniqueName: \"kubernetes.io/projected/ca4543f3-e757-4b16-978f-862868529523-kube-api-access-d7czq\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:30:36.058870 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.058775    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca4543f3-e757-4b16-978f-862868529523" containerID="701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430" exitCode=0
Sep 08 13:30:36.058870 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.058815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" event={"ID":"ca4543f3-e757-4b16-978f-862868529523","Type":"ContainerDied","Data":"701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430"}
Sep 08 13:30:36.058870 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.058838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"
Sep 08 13:30:36.058870 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.058873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf" event={"ID":"ca4543f3-e757-4b16-978f-862868529523","Type":"ContainerDied","Data":"dbe2944031e97b492ea7d84345fa8a3326b70bb5acf32b652979498b379068f2"}
Sep 08 13:30:36.059191 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.058899    2580 scope.go:117] "RemoveContainer" containerID="9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b"
Sep 08 13:30:36.059628 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.059602    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:30:36.067089 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.067075    2580 scope.go:117] "RemoveContainer" containerID="701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430"
Sep 08 13:30:36.074617 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.074593    2580 scope.go:117] "RemoveContainer" containerID="9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b"
Sep 08 13:30:36.074931 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:30:36.074899    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b\": container with ID starting with 9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b not found: ID does not exist" containerID="9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b"
Sep 08 13:30:36.075022 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.074983    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b"} err="failed to get container status \"9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b\": rpc error: code = NotFound desc = could not find container \"9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b\": container with ID starting with 9f6b494bfb6f816e0e8b109f76171d85799e991935953befd057d8e7e08e086b not found: ID does not exist"
Sep 08 13:30:36.075022 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.075011    2580 scope.go:117] "RemoveContainer" containerID="701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430"
Sep 08 13:30:36.075295 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:30:36.075280    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430\": container with ID starting with 701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430 not found: ID does not exist" containerID="701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430"
Sep 08 13:30:36.075350 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.075304    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430"} err="failed to get container status \"701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430\": rpc error: code = NotFound desc = could not find container \"701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430\": container with ID starting with 701b94467dae93f77b4507f784eea00e92263675e5cb8d9333972177fe14b430 not found: ID does not exist"
Sep 08 13:30:36.076350 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.076330    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"]
Sep 08 13:30:36.080284 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.080252    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f794f-predictor-75b57f85bc-4s5nf"]
Sep 08 13:30:36.742225 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:36.742199    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:30:37.750430 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:37.750382    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4543f3-e757-4b16-978f-862868529523" path="/var/lib/kubelet/pods/ca4543f3-e757-4b16-978f-862868529523/volumes"
Sep 08 13:30:41.063704 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:41.063676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:30:41.064224 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:41.064191    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:30:51.064095 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:51.064055    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:30:57.760665 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:57.760626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:58.273175 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:30:58.273145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:01.065123 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:01.065083    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:31:07.886498 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:07.886455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:07.992321 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:07.992279    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"]
Sep 08 13:31:07.992607 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:07.992566    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container" containerID="cri-o://a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d" gracePeriod=30
Sep 08 13:31:07.992698 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:07.992599    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kube-rbac-proxy" containerID="cri-o://aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac" gracePeriod=30
Sep 08 13:31:08.027361 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.027335    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"]
Sep 08 13:31:08.027606 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.027584    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container" containerID="cri-o://1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650" gracePeriod=30
Sep 08 13:31:08.027693 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.027616    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kube-rbac-proxy" containerID="cri-o://1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27" gracePeriod=30
Sep 08 13:31:08.044242 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.044210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:08.115829 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.115798    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"]
Sep 08 13:31:08.116361 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.116336    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container"
Sep 08 13:31:08.116361 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.116354    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container"
Sep 08 13:31:08.116361 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.116366    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kube-rbac-proxy"
Sep 08 13:31:08.116597 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.116375    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kube-rbac-proxy"
Sep 08 13:31:08.116597 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.116460    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kserve-container"
Sep 08 13:31:08.116597 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.116474    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4543f3-e757-4b16-978f-862868529523" containerName="kube-rbac-proxy"
Sep 08 13:31:08.117129 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.117100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:08.119180 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.119163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.125935 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.122016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-30f77-predictor-serving-cert\""
Sep 08 13:31:08.125935 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.122600    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-30f77-kube-rbac-proxy-sar-config\""
Sep 08 13:31:08.130180 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.130155    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"]
Sep 08 13:31:08.194609 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.194574    2580 generic.go:356] "Generic (PLEG): container finished" podID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerID="1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27" exitCode=2
Sep 08 13:31:08.194770 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.194647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" event={"ID":"83493665-53a4-4bed-8f89-0f2d1940bc18","Type":"ContainerDied","Data":"1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27"}
Sep 08 13:31:08.197183 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.197155    2580 generic.go:356] "Generic (PLEG): container finished" podID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerID="aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac" exitCode=2
Sep 08 13:31:08.197310 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.197202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" event={"ID":"9deb1efd-bcdf-40e2-9410-0a86324765a1","Type":"ContainerDied","Data":"aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac"}
Sep 08 13:31:08.215732 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.215684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:08.272545 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.272515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2e4e77-5919-4168-84f7-3d085eabbd6e-proxy-tls\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.272720 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.272588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw9sw\" (UniqueName: \"kubernetes.io/projected/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-api-access-zw9sw\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.272720 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.272612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.373300 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.373268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw9sw\" (UniqueName: \"kubernetes.io/projected/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-api-access-zw9sw\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.373483 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.373307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.373483 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.373377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2e4e77-5919-4168-84f7-3d085eabbd6e-proxy-tls\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.373602 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:31:08.373531    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-30f77-predictor-serving-cert: secret "success-200-isvc-30f77-predictor-serving-cert" not found
Sep 08 13:31:08.373602 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:31:08.373588    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a2e4e77-5919-4168-84f7-3d085eabbd6e-proxy-tls podName:2a2e4e77-5919-4168-84f7-3d085eabbd6e nodeName:}" failed. No retries permitted until 2026-09-08 13:31:08.873571775 +0000 UTC m=+881.573502399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a2e4e77-5919-4168-84f7-3d085eabbd6e-proxy-tls") pod "success-200-isvc-30f77-predictor-6b79cc6889-k52x5" (UID: "2a2e4e77-5919-4168-84f7-3d085eabbd6e") : secret "success-200-isvc-30f77-predictor-serving-cert" not found
Sep 08 13:31:08.374134 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.374110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.383230 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.383209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw9sw\" (UniqueName: \"kubernetes.io/projected/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-api-access-zw9sw\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.878998 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.878957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2e4e77-5919-4168-84f7-3d085eabbd6e-proxy-tls\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:08.881422 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:08.881383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2e4e77-5919-4168-84f7-3d085eabbd6e-proxy-tls\") pod \"success-200-isvc-30f77-predictor-6b79cc6889-k52x5\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") " pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:09.034604 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:09.034577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:09.304982 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:09.304888    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"]
Sep 08 13:31:09.310494 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:31:09.310454    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2e4e77_5919_4168_84f7_3d085eabbd6e.slice/crio-53b75151db556c349b592f3d9d503bbb0f22831a2fa248d27cb09e0fae2d77ca WatchSource:0}: Error finding container 53b75151db556c349b592f3d9d503bbb0f22831a2fa248d27cb09e0fae2d77ca: Status 404 returned error can't find the container with id 53b75151db556c349b592f3d9d503bbb0f22831a2fa248d27cb09e0fae2d77ca
Sep 08 13:31:09.775704 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:09.775657    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 08 13:31:09.863942 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:09.863907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:09.972238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:09.972197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:10.214899 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:10.214868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" event={"ID":"2a2e4e77-5919-4168-84f7-3d085eabbd6e","Type":"ContainerStarted","Data":"7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d"}
Sep 08 13:31:10.215323 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:10.214928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" event={"ID":"2a2e4e77-5919-4168-84f7-3d085eabbd6e","Type":"ContainerStarted","Data":"f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483"}
Sep 08 13:31:10.215323 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:10.214954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" event={"ID":"2a2e4e77-5919-4168-84f7-3d085eabbd6e","Type":"ContainerStarted","Data":"53b75151db556c349b592f3d9d503bbb0f22831a2fa248d27cb09e0fae2d77ca"}
Sep 08 13:31:10.215323 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:10.215072    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:10.215323 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:10.215223    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:10.216463 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:10.216439    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:31:10.235312 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:10.235265    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podStartSLOduration=2.235251838 podStartE2EDuration="2.235251838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:31:10.233754255 +0000 UTC m=+882.933684881" watchObservedRunningTime="2026-09-08 13:31:10.235251838 +0000 UTC m=+882.935182472"
Sep 08 13:31:11.065038 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:11.064998    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:31:11.219029 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:11.218992    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:31:11.734147 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:11.734119    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 08 13:31:11.864736 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:11.864710    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:31:12.004832 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.004727    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-cabundle-cert\") pod \"83493665-53a4-4bed-8f89-0f2d1940bc18\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") "
Sep 08 13:31:12.004989 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.004833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-rbac-proxy-sar-config\") pod \"83493665-53a4-4bed-8f89-0f2d1940bc18\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") "
Sep 08 13:31:12.004989 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.004884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83493665-53a4-4bed-8f89-0f2d1940bc18-proxy-tls\") pod \"83493665-53a4-4bed-8f89-0f2d1940bc18\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") "
Sep 08 13:31:12.004989 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.004929    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83493665-53a4-4bed-8f89-0f2d1940bc18-kserve-provision-location\") pod \"83493665-53a4-4bed-8f89-0f2d1940bc18\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") "
Sep 08 13:31:12.004989 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.004952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkx7s\" (UniqueName: \"kubernetes.io/projected/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-api-access-jkx7s\") pod \"83493665-53a4-4bed-8f89-0f2d1940bc18\" (UID: \"83493665-53a4-4bed-8f89-0f2d1940bc18\") "
Sep 08 13:31:12.005282 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.005169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83493665-53a4-4bed-8f89-0f2d1940bc18" (UID: "83493665-53a4-4bed-8f89-0f2d1940bc18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:12.005344 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.005166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83493665-53a4-4bed-8f89-0f2d1940bc18" (UID: "83493665-53a4-4bed-8f89-0f2d1940bc18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:12.005344 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.005315    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83493665-53a4-4bed-8f89-0f2d1940bc18-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83493665-53a4-4bed-8f89-0f2d1940bc18" (UID: "83493665-53a4-4bed-8f89-0f2d1940bc18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:31:12.007380 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.007350    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-api-access-jkx7s" (OuterVolumeSpecName: "kube-api-access-jkx7s") pod "83493665-53a4-4bed-8f89-0f2d1940bc18" (UID: "83493665-53a4-4bed-8f89-0f2d1940bc18"). InnerVolumeSpecName "kube-api-access-jkx7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:31:12.007721 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.007693    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83493665-53a4-4bed-8f89-0f2d1940bc18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83493665-53a4-4bed-8f89-0f2d1940bc18" (UID: "83493665-53a4-4bed-8f89-0f2d1940bc18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:31:12.106203 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.106165    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83493665-53a4-4bed-8f89-0f2d1940bc18-kserve-provision-location\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:12.106203 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.106194    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkx7s\" (UniqueName: \"kubernetes.io/projected/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-api-access-jkx7s\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:12.106203 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.106206    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-cabundle-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:12.106203 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.106214    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83493665-53a4-4bed-8f89-0f2d1940bc18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:12.106584 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.106223    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83493665-53a4-4bed-8f89-0f2d1940bc18-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:12.230846 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.230795    2580 generic.go:356] "Generic (PLEG): container finished" podID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerID="1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650" exitCode=0
Sep 08 13:31:12.231201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.230967    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"
Sep 08 13:31:12.231201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.230979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" event={"ID":"83493665-53a4-4bed-8f89-0f2d1940bc18","Type":"ContainerDied","Data":"1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650"}
Sep 08 13:31:12.231201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.231082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2" event={"ID":"83493665-53a4-4bed-8f89-0f2d1940bc18","Type":"ContainerDied","Data":"28baf33fc9927e97d9085269473282fd7c30cdd162e55cc91ee18609c740b3f3"}
Sep 08 13:31:12.231201 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.231107    2580 scope.go:117] "RemoveContainer" containerID="1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27"
Sep 08 13:31:12.240584 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.240567    2580 scope.go:117] "RemoveContainer" containerID="1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650"
Sep 08 13:31:12.248264 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.248245    2580 scope.go:117] "RemoveContainer" containerID="d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797"
Sep 08 13:31:12.253044 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.253020    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"]
Sep 08 13:31:12.255599 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.255582    2580 scope.go:117] "RemoveContainer" containerID="1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27"
Sep 08 13:31:12.255878 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:31:12.255852    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27\": container with ID starting with 1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27 not found: ID does not exist" containerID="1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27"
Sep 08 13:31:12.255951 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.255889    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27"} err="failed to get container status \"1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27\": rpc error: code = NotFound desc = could not find container \"1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27\": container with ID starting with 1afa59296601d1db6c8097a8fb30c80b15aeaf9d8516650e73b1a45a818acf27 not found: ID does not exist"
Sep 08 13:31:12.255951 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.255912    2580 scope.go:117] "RemoveContainer" containerID="1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650"
Sep 08 13:31:12.256175 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:31:12.256156    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650\": container with ID starting with 1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650 not found: ID does not exist" containerID="1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650"
Sep 08 13:31:12.256238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.256183    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650"} err="failed to get container status \"1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650\": rpc error: code = NotFound desc = could not find container \"1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650\": container with ID starting with 1ec40623ffb3f90bd7e8f48b7158120bf39c18b9b7218e5c400419c70af1e650 not found: ID does not exist"
Sep 08 13:31:12.256238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.256205    2580 scope.go:117] "RemoveContainer" containerID="d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797"
Sep 08 13:31:12.256645 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:31:12.256617    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797\": container with ID starting with d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797 not found: ID does not exist" containerID="d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797"
Sep 08 13:31:12.256745 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.256650    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797"} err="failed to get container status \"d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797\": rpc error: code = NotFound desc = could not find container \"d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797\": container with ID starting with d032f4ec7925f3995e497878379613879226c508cce2ca9761ff5ddc08cd9797 not found: ID does not exist"
Sep 08 13:31:12.258435 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:12.258414    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-d85h2"]
Sep 08 13:31:13.134088 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.134059    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:31:13.216214 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216140    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-rbac-proxy-sar-config\") pod \"9deb1efd-bcdf-40e2-9410-0a86324765a1\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") "
Sep 08 13:31:13.216214 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-cabundle-cert\") pod \"9deb1efd-bcdf-40e2-9410-0a86324765a1\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") "
Sep 08 13:31:13.216416 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216220    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9deb1efd-bcdf-40e2-9410-0a86324765a1-kserve-provision-location\") pod \"9deb1efd-bcdf-40e2-9410-0a86324765a1\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") "
Sep 08 13:31:13.216416 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9deb1efd-bcdf-40e2-9410-0a86324765a1-proxy-tls\") pod \"9deb1efd-bcdf-40e2-9410-0a86324765a1\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") "
Sep 08 13:31:13.216536 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5vkg\" (UniqueName: \"kubernetes.io/projected/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-api-access-q5vkg\") pod \"9deb1efd-bcdf-40e2-9410-0a86324765a1\" (UID: \"9deb1efd-bcdf-40e2-9410-0a86324765a1\") "
Sep 08 13:31:13.216536 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216491    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deb1efd-bcdf-40e2-9410-0a86324765a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9deb1efd-bcdf-40e2-9410-0a86324765a1" (UID: "9deb1efd-bcdf-40e2-9410-0a86324765a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:31:13.216639 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216598    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9deb1efd-bcdf-40e2-9410-0a86324765a1" (UID: "9deb1efd-bcdf-40e2-9410-0a86324765a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:13.216639 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9deb1efd-bcdf-40e2-9410-0a86324765a1" (UID: "9deb1efd-bcdf-40e2-9410-0a86324765a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:13.216740 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216731    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-cabundle-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:13.216777 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216745    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9deb1efd-bcdf-40e2-9410-0a86324765a1-kserve-provision-location\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:13.216777 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.216754    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:13.218443 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.218416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9deb1efd-bcdf-40e2-9410-0a86324765a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9deb1efd-bcdf-40e2-9410-0a86324765a1" (UID: "9deb1efd-bcdf-40e2-9410-0a86324765a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:31:13.218533 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.218513    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-api-access-q5vkg" (OuterVolumeSpecName: "kube-api-access-q5vkg") pod "9deb1efd-bcdf-40e2-9410-0a86324765a1" (UID: "9deb1efd-bcdf-40e2-9410-0a86324765a1"). InnerVolumeSpecName "kube-api-access-q5vkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:31:13.238509 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.238478    2580 generic.go:356] "Generic (PLEG): container finished" podID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerID="a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d" exitCode=0
Sep 08 13:31:13.238923 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.238515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" event={"ID":"9deb1efd-bcdf-40e2-9410-0a86324765a1","Type":"ContainerDied","Data":"a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d"}
Sep 08 13:31:13.238923 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.238540    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"
Sep 08 13:31:13.238923 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.238564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9" event={"ID":"9deb1efd-bcdf-40e2-9410-0a86324765a1","Type":"ContainerDied","Data":"52c406bc14c092f5b414434311420e4d199430f9ddfb910aa6d27a76acabb26b"}
Sep 08 13:31:13.238923 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.238589    2580 scope.go:117] "RemoveContainer" containerID="aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac"
Sep 08 13:31:13.246704 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.246686    2580 scope.go:117] "RemoveContainer" containerID="a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d"
Sep 08 13:31:13.253907 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.253891    2580 scope.go:117] "RemoveContainer" containerID="bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121"
Sep 08 13:31:13.261144 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.261123    2580 scope.go:117] "RemoveContainer" containerID="aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac"
Sep 08 13:31:13.261473 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:31:13.261453    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac\": container with ID starting with aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac not found: ID does not exist" containerID="aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac"
Sep 08 13:31:13.261563 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.261483    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac"} err="failed to get container status \"aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac\": rpc error: code = NotFound desc = could not find container \"aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac\": container with ID starting with aa212aeac44c74a4b89aeefd1e55c816ab68766cf5f4ed74aafd17dd74d3d9ac not found: ID does not exist"
Sep 08 13:31:13.261563 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.261506    2580 scope.go:117] "RemoveContainer" containerID="a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d"
Sep 08 13:31:13.261809 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:31:13.261787    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d\": container with ID starting with a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d not found: ID does not exist" containerID="a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d"
Sep 08 13:31:13.261848 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.261820    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d"} err="failed to get container status \"a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d\": rpc error: code = NotFound desc = could not find container \"a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d\": container with ID starting with a2c090190e5b48880cbae177a9fac897e704fcacb475420aafd5fa37a7353b6d not found: ID does not exist"
Sep 08 13:31:13.261848 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.261841    2580 scope.go:117] "RemoveContainer" containerID="bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121"
Sep 08 13:31:13.262115 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:31:13.262097    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121\": container with ID starting with bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121 not found: ID does not exist" containerID="bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121"
Sep 08 13:31:13.262167 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.262120    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121"} err="failed to get container status \"bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121\": rpc error: code = NotFound desc = could not find container \"bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121\": container with ID starting with bc1f21111725b4a24dc93252769c6f80ccfd30a34e359c2de65576802b745121 not found: ID does not exist"
Sep 08 13:31:13.262550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.262534    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"]
Sep 08 13:31:13.266298 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.266278    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7f476ffc7-bw8k9"]
Sep 08 13:31:13.317952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.317925    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9deb1efd-bcdf-40e2-9410-0a86324765a1-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:13.317952 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.317949    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5vkg\" (UniqueName: \"kubernetes.io/projected/9deb1efd-bcdf-40e2-9410-0a86324765a1-kube-api-access-q5vkg\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:31:13.750252 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.750218    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" path="/var/lib/kubelet/pods/83493665-53a4-4bed-8f89-0f2d1940bc18/volumes"
Sep 08 13:31:13.750708 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:13.750694    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" path="/var/lib/kubelet/pods/9deb1efd-bcdf-40e2-9410-0a86324765a1/volumes"
Sep 08 13:31:16.223939 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:16.223910    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:31:16.224377 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:16.224353    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:31:21.065033 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:21.065000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:31:26.224720 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:26.224674    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:31:32.200558 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:32.200522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:32.423740 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:32.423700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:36.224493 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:36.224440    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:31:46.224720 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:46.224682    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:31:56.225149 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:31:56.225119    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:32:08.048505 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:32:08.048459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:10.920775 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:32:10.920742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:59.734828 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:32:59.734728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:00.586194 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:33:00.586157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:44.105987 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:33:44.105953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:44.176033 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:33:44.175991    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:17.396108 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:35:17.396066    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:29.316297 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:35:29.316260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:29.378096 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:35:29.378060    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:47.225151 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225118    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"]
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225536    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="storage-initializer"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225554    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="storage-initializer"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225565    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225571    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225582    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kube-rbac-proxy"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225589    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kube-rbac-proxy"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225595    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225600    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225609    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kube-rbac-proxy"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225617    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kube-rbac-proxy"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225628    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="storage-initializer"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225634    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="storage-initializer"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225690    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kube-rbac-proxy"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225697    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kube-rbac-proxy"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225705    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9deb1efd-bcdf-40e2-9410-0a86324765a1" containerName="kserve-container"
Sep 08 13:39:47.229358 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.225712    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="83493665-53a4-4bed-8f89-0f2d1940bc18" containerName="kserve-container"
Sep 08 13:39:47.229911 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.229885    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"]
Sep 08 13:39:47.230070 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.229979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.230213 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.230173    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container" containerID="cri-o://6858f1ac9898921f877845d46990e84169608410fe5d02dcbc53d04659e72610" gracePeriod=30
Sep 08 13:39:47.230294 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.230221    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kube-rbac-proxy" containerID="cri-o://0bf11a7051792defaa5a5e0152f5782f9462d13b6b4122c6f292e9f3f48a3a9d" gracePeriod=30
Sep 08 13:39:47.232615 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.232591    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-601fd-kube-rbac-proxy-sar-config\""
Sep 08 13:39:47.232740 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.232633    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-601fd-predictor-serving-cert\""
Sep 08 13:39:47.243518 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.243497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"]
Sep 08 13:39:47.321567 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.321516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt52n\" (UniqueName: \"kubernetes.io/projected/4bc8c230-e449-49d1-b580-5687be1198f6-kube-api-access-vt52n\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.321757 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.321643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc8c230-e449-49d1-b580-5687be1198f6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.321757 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.321688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc8c230-e449-49d1-b580-5687be1198f6-proxy-tls\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.422978 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.422934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt52n\" (UniqueName: \"kubernetes.io/projected/4bc8c230-e449-49d1-b580-5687be1198f6-kube-api-access-vt52n\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.423185 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.423016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc8c230-e449-49d1-b580-5687be1198f6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.423185 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.423048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc8c230-e449-49d1-b580-5687be1198f6-proxy-tls\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.423303 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:39:47.423185    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-601fd-predictor-serving-cert: secret "success-200-isvc-601fd-predictor-serving-cert" not found
Sep 08 13:39:47.423303 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:39:47.423247    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bc8c230-e449-49d1-b580-5687be1198f6-proxy-tls podName:4bc8c230-e449-49d1-b580-5687be1198f6 nodeName:}" failed. No retries permitted until 2026-09-08 13:39:47.923227073 +0000 UTC m=+1400.623157691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4bc8c230-e449-49d1-b580-5687be1198f6-proxy-tls") pod "success-200-isvc-601fd-predictor-646cfc64df-ckjl2" (UID: "4bc8c230-e449-49d1-b580-5687be1198f6") : secret "success-200-isvc-601fd-predictor-serving-cert" not found
Sep 08 13:39:47.424074 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.424045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc8c230-e449-49d1-b580-5687be1198f6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.433567 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.433540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt52n\" (UniqueName: \"kubernetes.io/projected/4bc8c230-e449-49d1-b580-5687be1198f6-kube-api-access-vt52n\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.927192 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.927141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc8c230-e449-49d1-b580-5687be1198f6-proxy-tls\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:47.929529 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:47.929505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc8c230-e449-49d1-b580-5687be1198f6-proxy-tls\") pod \"success-200-isvc-601fd-predictor-646cfc64df-ckjl2\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") " pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:48.151414 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:48.151369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:48.342683 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:48.342644    2580 generic.go:356] "Generic (PLEG): container finished" podID="7cda03f3-9df3-4741-8799-8874f44370b4" containerID="0bf11a7051792defaa5a5e0152f5782f9462d13b6b4122c6f292e9f3f48a3a9d" exitCode=2
Sep 08 13:39:48.343043 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:48.342718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" event={"ID":"7cda03f3-9df3-4741-8799-8874f44370b4","Type":"ContainerDied","Data":"0bf11a7051792defaa5a5e0152f5782f9462d13b6b4122c6f292e9f3f48a3a9d"}
Sep 08 13:39:48.425277 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:48.425199    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"]
Sep 08 13:39:48.427735 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:39:48.427706    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc8c230_e449_49d1_b580_5687be1198f6.slice/crio-80f7fef3eff1310d385b3f3f693dde2dcfdab290f0bdb95f36a9aace69fd743b WatchSource:0}: Error finding container 80f7fef3eff1310d385b3f3f693dde2dcfdab290f0bdb95f36a9aace69fd743b: Status 404 returned error can't find the container with id 80f7fef3eff1310d385b3f3f693dde2dcfdab290f0bdb95f36a9aace69fd743b
Sep 08 13:39:49.348683 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:49.348647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" event={"ID":"4bc8c230-e449-49d1-b580-5687be1198f6","Type":"ContainerStarted","Data":"de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89"}
Sep 08 13:39:49.349060 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:49.348702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" event={"ID":"4bc8c230-e449-49d1-b580-5687be1198f6","Type":"ContainerStarted","Data":"19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf"}
Sep 08 13:39:49.349060 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:49.348722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" event={"ID":"4bc8c230-e449-49d1-b580-5687be1198f6","Type":"ContainerStarted","Data":"80f7fef3eff1310d385b3f3f693dde2dcfdab290f0bdb95f36a9aace69fd743b"}
Sep 08 13:39:49.349060 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:49.348744    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:49.368821 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:49.368766    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" podStartSLOduration=2.368750328 podStartE2EDuration="2.368750328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:39:49.36752417 +0000 UTC m=+1402.067454803" watchObservedRunningTime="2026-09-08 13:39:49.368750328 +0000 UTC m=+1402.068680962"
Sep 08 13:39:49.899363 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:49.899329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:50.355046 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.355016    2580 generic.go:356] "Generic (PLEG): container finished" podID="7cda03f3-9df3-4741-8799-8874f44370b4" containerID="6858f1ac9898921f877845d46990e84169608410fe5d02dcbc53d04659e72610" exitCode=0
Sep 08 13:39:50.355383 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.355083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" event={"ID":"7cda03f3-9df3-4741-8799-8874f44370b4","Type":"ContainerDied","Data":"6858f1ac9898921f877845d46990e84169608410fe5d02dcbc53d04659e72610"}
Sep 08 13:39:50.355465 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.355422    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:50.356981 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.356954    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:39:50.473430 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.473388    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:39:50.651677 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.651640    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls\") pod \"7cda03f3-9df3-4741-8799-8874f44370b4\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") "
Sep 08 13:39:50.651873 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.651714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cda03f3-9df3-4741-8799-8874f44370b4-kube-rbac-proxy-sar-config\") pod \"7cda03f3-9df3-4741-8799-8874f44370b4\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") "
Sep 08 13:39:50.651873 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.651744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2s46\" (UniqueName: \"kubernetes.io/projected/7cda03f3-9df3-4741-8799-8874f44370b4-kube-api-access-s2s46\") pod \"7cda03f3-9df3-4741-8799-8874f44370b4\" (UID: \"7cda03f3-9df3-4741-8799-8874f44370b4\") "
Sep 08 13:39:50.652077 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.652054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cda03f3-9df3-4741-8799-8874f44370b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7cda03f3-9df3-4741-8799-8874f44370b4" (UID: "7cda03f3-9df3-4741-8799-8874f44370b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:39:50.653924 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.653898    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cda03f3-9df3-4741-8799-8874f44370b4" (UID: "7cda03f3-9df3-4741-8799-8874f44370b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:39:50.654026 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.653919    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cda03f3-9df3-4741-8799-8874f44370b4-kube-api-access-s2s46" (OuterVolumeSpecName: "kube-api-access-s2s46") pod "7cda03f3-9df3-4741-8799-8874f44370b4" (UID: "7cda03f3-9df3-4741-8799-8874f44370b4"). InnerVolumeSpecName "kube-api-access-s2s46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:39:50.753337 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.753303    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cda03f3-9df3-4741-8799-8874f44370b4-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:39:50.753337 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.753331    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cda03f3-9df3-4741-8799-8874f44370b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:39:50.753337 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:50.753343    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2s46\" (UniqueName: \"kubernetes.io/projected/7cda03f3-9df3-4741-8799-8874f44370b4-kube-api-access-s2s46\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:39:51.360374 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:51.360347    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"
Sep 08 13:39:51.360827 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:51.360382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2" event={"ID":"7cda03f3-9df3-4741-8799-8874f44370b4","Type":"ContainerDied","Data":"b2aee55d843a809dcbeaa8d6b743eb3789921d9635bcc054b2e2c2e8cbfaf46f"}
Sep 08 13:39:51.360827 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:51.360456    2580 scope.go:117] "RemoveContainer" containerID="0bf11a7051792defaa5a5e0152f5782f9462d13b6b4122c6f292e9f3f48a3a9d"
Sep 08 13:39:51.360827 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:51.360785    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:39:51.369230 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:51.369215    2580 scope.go:117] "RemoveContainer" containerID="6858f1ac9898921f877845d46990e84169608410fe5d02dcbc53d04659e72610"
Sep 08 13:39:51.383267 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:51.383247    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"]
Sep 08 13:39:51.388748 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:51.388726    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87be3-predictor-845b844549-cpcg2"]
Sep 08 13:39:51.525730 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:51.525695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:51.751010 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:51.750921    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" path="/var/lib/kubelet/pods/7cda03f3-9df3-4741-8799-8874f44370b4/volumes"
Sep 08 13:39:56.365607 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:56.365576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:39:56.366188 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:39:56.366158    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:40:04.036850 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:04.036807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:06.366842 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:06.366795    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:40:16.366505 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:16.366464    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:40:22.692635 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:22.692564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:22.819174 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:22.819141    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"]
Sep 08 13:40:22.819418 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:22.819376    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container" containerID="cri-o://f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483" gracePeriod=30
Sep 08 13:40:22.819510 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:22.819424    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kube-rbac-proxy" containerID="cri-o://7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d" gracePeriod=30
Sep 08 13:40:22.861330 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:22.861279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:22.992607 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:22.992517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:23.058712 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.058678    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"]
Sep 08 13:40:23.059041 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.059025    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container"
Sep 08 13:40:23.059102 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.059043    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container"
Sep 08 13:40:23.059102 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.059065    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kube-rbac-proxy"
Sep 08 13:40:23.059102 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.059070    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kube-rbac-proxy"
Sep 08 13:40:23.059215 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.059124    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kserve-container"
Sep 08 13:40:23.059215 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.059135    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cda03f3-9df3-4741-8799-8874f44370b4" containerName="kube-rbac-proxy"
Sep 08 13:40:23.062227 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.062212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:23.064810 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:23.064782    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"success-200-isvc-e90df-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-140-239.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-239.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e90df-predictor-serving-cert\"" type="*v1.Secret"
Sep 08 13:40:23.064926 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:23.064828    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"success-200-isvc-e90df-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-140-239.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-239.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e90df-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 08 13:40:23.064992 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:23.064970    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" is forbidden: User \"system:node:ip-10-0-140-239.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-239.ec2.internal' and this object" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:23.091042 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.091017    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"]
Sep 08 13:40:23.108619 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.108590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:23.122978 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.122948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2shs\" (UniqueName: \"kubernetes.io/projected/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-api-access-h2shs\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:23.123129 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.123007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:23.123129 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.123078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d5a9db-31ea-4c49-925e-5910db3e30da-proxy-tls\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:23.216080 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.216039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:23.223681 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.223646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d5a9db-31ea-4c49-925e-5910db3e30da-proxy-tls\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:23.223860 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.223833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2shs\" (UniqueName: \"kubernetes.io/projected/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-api-access-h2shs\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:23.223925 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.223894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:23.236030 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.235995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2shs\" (UniqueName: \"kubernetes.io/projected/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-api-access-h2shs\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:23.372765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.372727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:23.496760 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.496728    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerID="7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d" exitCode=2
Sep 08 13:40:23.496931 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.496801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" event={"ID":"2a2e4e77-5919-4168-84f7-3d085eabbd6e","Type":"ContainerDied","Data":"7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d"}
Sep 08 13:40:23.992628 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:23.992591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:24.223891 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:24.223857    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e90df-predictor-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 08 13:40:24.224072 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:24.223940    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5d5a9db-31ea-4c49-925e-5910db3e30da-proxy-tls podName:e5d5a9db-31ea-4c49-925e-5910db3e30da nodeName:}" failed. No retries permitted until 2026-09-08 13:40:24.723920752 +0000 UTC m=+1437.423851364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e5d5a9db-31ea-4c49-925e-5910db3e30da-proxy-tls") pod "success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" (UID: "e5d5a9db-31ea-4c49-925e-5910db3e30da") : failed to sync secret cache: timed out waiting for the condition
Sep 08 13:40:24.224072 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:24.224018    2580 configmap.go:193] Couldn't get configMap kserve-ci-e2e-test/success-200-isvc-e90df-kube-rbac-proxy-sar-config: failed to sync configmap cache: timed out waiting for the condition
Sep 08 13:40:24.224173 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:24.224091    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-rbac-proxy-sar-config podName:e5d5a9db-31ea-4c49-925e-5910db3e30da nodeName:}" failed. No retries permitted until 2026-09-08 13:40:24.724076552 +0000 UTC m=+1437.424007169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" (UniqueName: "kubernetes.io/configmap/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-rbac-proxy-sar-config") pod "success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" (UID: "e5d5a9db-31ea-4c49-925e-5910db3e30da") : failed to sync configmap cache: timed out waiting for the condition
Sep 08 13:40:24.287552 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:24.287468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e90df-kube-rbac-proxy-sar-config\""
Sep 08 13:40:24.423652 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:24.423617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e90df-predictor-serving-cert\""
Sep 08 13:40:24.736998 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:24.736962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:24.737163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:24.737019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d5a9db-31ea-4c49-925e-5910db3e30da-proxy-tls\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:24.737639 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:24.737612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:24.739438 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:24.739420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d5a9db-31ea-4c49-925e-5910db3e30da-proxy-tls\") pod \"success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") " pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:24.874482 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:24.874441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:25.156121 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:25.156010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"]
Sep 08 13:40:25.158962 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:40:25.158935    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d5a9db_31ea_4c49_925e_5910db3e30da.slice/crio-3b4ddbbd94db5eee2cf184a3906c06310bfeccf65f93807af2eb25b461008a9a WatchSource:0}: Error finding container 3b4ddbbd94db5eee2cf184a3906c06310bfeccf65f93807af2eb25b461008a9a: Status 404 returned error can't find the container with id 3b4ddbbd94db5eee2cf184a3906c06310bfeccf65f93807af2eb25b461008a9a
Sep 08 13:40:25.511459 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:25.511169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" event={"ID":"e5d5a9db-31ea-4c49-925e-5910db3e30da","Type":"ContainerStarted","Data":"f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016"}
Sep 08 13:40:25.511459 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:25.511240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" event={"ID":"e5d5a9db-31ea-4c49-925e-5910db3e30da","Type":"ContainerStarted","Data":"08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490"}
Sep 08 13:40:25.511459 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:25.511270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" event={"ID":"e5d5a9db-31ea-4c49-925e-5910db3e30da","Type":"ContainerStarted","Data":"3b4ddbbd94db5eee2cf184a3906c06310bfeccf65f93807af2eb25b461008a9a"}
Sep 08 13:40:25.511459 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:25.511302    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:25.542930 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:25.542864    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" podStartSLOduration=2.542845569 podStartE2EDuration="2.542845569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:40:25.540777264 +0000 UTC m=+1438.240707898" watchObservedRunningTime="2026-09-08 13:40:25.542845569 +0000 UTC m=+1438.242776202"
Sep 08 13:40:26.267435 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.267389    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:40:26.351078 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.351038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-rbac-proxy-sar-config\") pod \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") "
Sep 08 13:40:26.351278 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.351109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw9sw\" (UniqueName: \"kubernetes.io/projected/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-api-access-zw9sw\") pod \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") "
Sep 08 13:40:26.351278 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.351153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2e4e77-5919-4168-84f7-3d085eabbd6e-proxy-tls\") pod \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\" (UID: \"2a2e4e77-5919-4168-84f7-3d085eabbd6e\") "
Sep 08 13:40:26.351525 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.351499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a2e4e77-5919-4168-84f7-3d085eabbd6e" (UID: "2a2e4e77-5919-4168-84f7-3d085eabbd6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:26.353288 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.353265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2e4e77-5919-4168-84f7-3d085eabbd6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a2e4e77-5919-4168-84f7-3d085eabbd6e" (UID: "2a2e4e77-5919-4168-84f7-3d085eabbd6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:40:26.353392 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.353383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-api-access-zw9sw" (OuterVolumeSpecName: "kube-api-access-zw9sw") pod "2a2e4e77-5919-4168-84f7-3d085eabbd6e" (UID: "2a2e4e77-5919-4168-84f7-3d085eabbd6e"). InnerVolumeSpecName "kube-api-access-zw9sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:40:26.366242 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.366207    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:40:26.451906 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.451864    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:40:26.451906 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.451900    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw9sw\" (UniqueName: \"kubernetes.io/projected/2a2e4e77-5919-4168-84f7-3d085eabbd6e-kube-api-access-zw9sw\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:40:26.451906 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.451911    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2e4e77-5919-4168-84f7-3d085eabbd6e-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:40:26.516500 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.516463    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerID="f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483" exitCode=0
Sep 08 13:40:26.516666 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.516536    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"
Sep 08 13:40:26.516666 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.516547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" event={"ID":"2a2e4e77-5919-4168-84f7-3d085eabbd6e","Type":"ContainerDied","Data":"f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483"}
Sep 08 13:40:26.516666 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.516630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" event={"ID":"2a2e4e77-5919-4168-84f7-3d085eabbd6e","Type":"ContainerDied","Data":"53b75151db556c349b592f3d9d503bbb0f22831a2fa248d27cb09e0fae2d77ca"}
Sep 08 13:40:26.516666 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.516657    2580 scope.go:117] "RemoveContainer" containerID="7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d"
Sep 08 13:40:26.517128 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.517107    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:26.518734 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.518707    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 08 13:40:26.525753 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.525733    2580 scope.go:117] "RemoveContainer" containerID="f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483"
Sep 08 13:40:26.533190 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.533174    2580 scope.go:117] "RemoveContainer" containerID="7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d"
Sep 08 13:40:26.533460 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:26.533439    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d\": container with ID starting with 7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d not found: ID does not exist" containerID="7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d"
Sep 08 13:40:26.533514 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.533468    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d"} err="failed to get container status \"7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d\": rpc error: code = NotFound desc = could not find container \"7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d\": container with ID starting with 7b82edfae40e41d47aa069c093e3f8451729b29631f2b3d0f410c561b1735a9d not found: ID does not exist"
Sep 08 13:40:26.533514 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.533484    2580 scope.go:117] "RemoveContainer" containerID="f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483"
Sep 08 13:40:26.533699 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:26.533683    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483\": container with ID starting with f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483 not found: ID does not exist" containerID="f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483"
Sep 08 13:40:26.533746 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.533703    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483"} err="failed to get container status \"f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483\": rpc error: code = NotFound desc = could not find container \"f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483\": container with ID starting with f7615717d2d2fbc1b1d8b868a4a4ca2d0137d9118914870a5e8b96b53e952483 not found: ID does not exist"
Sep 08 13:40:26.539427 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.539392    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"]
Sep 08 13:40:26.546820 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:26.546801    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5"]
Sep 08 13:40:27.220648 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:27.220594    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 08 13:40:27.225778 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:27.225742    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30f77-predictor-6b79cc6889-k52x5" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: i/o timeout"
Sep 08 13:40:27.521329 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:27.521242    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 08 13:40:27.749918 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:27.749882    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" path="/var/lib/kubelet/pods/2a2e4e77-5919-4168-84f7-3d085eabbd6e/volumes"
Sep 08 13:40:32.525864 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:32.525834    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:40:32.526342 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:32.526315    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 08 13:40:36.367461 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:36.367416    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:40:42.527090 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:42.527048    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 08 13:40:47.219687 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:47.219645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:47.568415 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:47.568318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:52.527258 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:52.527213    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 08 13:40:57.283528 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.283492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:57.365333 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.365294    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"]
Sep 08 13:40:57.365579 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.365544    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container" containerID="cri-o://19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf" gracePeriod=30
Sep 08 13:40:57.365721 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.365596    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kube-rbac-proxy" containerID="cri-o://de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89" gracePeriod=30
Sep 08 13:40:57.404589 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.404555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:57.504819 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.504777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:57.512567 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.512538    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"]
Sep 08 13:40:57.513091 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.513073    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container"
Sep 08 13:40:57.513091 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.513092    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container"
Sep 08 13:40:57.513221 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.513104    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kube-rbac-proxy"
Sep 08 13:40:57.513221 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.513114    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kube-rbac-proxy"
Sep 08 13:40:57.513221 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.513179    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kserve-container"
Sep 08 13:40:57.513221 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.513190    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2e4e77-5919-4168-84f7-3d085eabbd6e" containerName="kube-rbac-proxy"
Sep 08 13:40:57.522795 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.522770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:57.524912 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.524891    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-64898-predictor-serving-cert\""
Sep 08 13:40:57.525145 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.525128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-64898-kube-rbac-proxy-sar-config\""
Sep 08 13:40:57.527891 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.527867    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"]
Sep 08 13:40:57.609616 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.609577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:57.621048 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.621021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v7bf\" (UniqueName: \"kubernetes.io/projected/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-api-access-5v7bf\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:57.621225 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.621093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:57.621225 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.621157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c6a880-7d55-497a-8b8e-f3f62c7562cd-proxy-tls\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:57.643342 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.643309    2580 generic.go:356] "Generic (PLEG): container finished" podID="4bc8c230-e449-49d1-b580-5687be1198f6" containerID="de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89" exitCode=2
Sep 08 13:40:57.643505 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.643372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" event={"ID":"4bc8c230-e449-49d1-b580-5687be1198f6","Type":"ContainerDied","Data":"de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89"}
Sep 08 13:40:57.705615 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.705566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:57.721602 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.721573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:57.721753 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.721667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c6a880-7d55-497a-8b8e-f3f62c7562cd-proxy-tls\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:57.721753 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.721700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7bf\" (UniqueName: \"kubernetes.io/projected/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-api-access-5v7bf\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:57.721879 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:57.721828    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-64898-predictor-serving-cert: secret "success-200-isvc-64898-predictor-serving-cert" not found
Sep 08 13:40:57.721933 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:40:57.721914    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68c6a880-7d55-497a-8b8e-f3f62c7562cd-proxy-tls podName:68c6a880-7d55-497a-8b8e-f3f62c7562cd nodeName:}" failed. No retries permitted until 2026-09-08 13:40:58.221898957 +0000 UTC m=+1470.921829569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/68c6a880-7d55-497a-8b8e-f3f62c7562cd-proxy-tls") pod "success-200-isvc-64898-predictor-57884f876f-8wq7r" (UID: "68c6a880-7d55-497a-8b8e-f3f62c7562cd") : secret "success-200-isvc-64898-predictor-serving-cert" not found
Sep 08 13:40:57.722308 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.722288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:57.731765 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.731743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7bf\" (UniqueName: \"kubernetes.io/projected/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-api-access-5v7bf\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:57.824386 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.824349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:57.927372 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:57.927281    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:58.227460 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:58.227365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c6a880-7d55-497a-8b8e-f3f62c7562cd-proxy-tls\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:58.229716 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:58.229693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c6a880-7d55-497a-8b8e-f3f62c7562cd-proxy-tls\") pod \"success-200-isvc-64898-predictor-57884f876f-8wq7r\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") " pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:58.434764 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:58.434729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:58.724989 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:58.724963    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"]
Sep 08 13:40:58.727706 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:40:58.727677    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c6a880_7d55_497a_8b8e_f3f62c7562cd.slice/crio-6e8f15e911dfee2221708fa8ed31da78b9d2863493c4808de6cbe600659d6f83 WatchSource:0}: Error finding container 6e8f15e911dfee2221708fa8ed31da78b9d2863493c4808de6cbe600659d6f83: Status 404 returned error can't find the container with id 6e8f15e911dfee2221708fa8ed31da78b9d2863493c4808de6cbe600659d6f83
Sep 08 13:40:59.455203 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:59.455167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:59.557111 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:59.557072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:59.652533 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:59.652502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" event={"ID":"68c6a880-7d55-497a-8b8e-f3f62c7562cd","Type":"ContainerStarted","Data":"ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff"}
Sep 08 13:40:59.652732 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:59.652565    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:59.652732 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:59.652595    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:40:59.652732 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:59.652616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" event={"ID":"68c6a880-7d55-497a-8b8e-f3f62c7562cd","Type":"ContainerStarted","Data":"d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222"}
Sep 08 13:40:59.652732 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:59.652633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" event={"ID":"68c6a880-7d55-497a-8b8e-f3f62c7562cd","Type":"ContainerStarted","Data":"6e8f15e911dfee2221708fa8ed31da78b9d2863493c4808de6cbe600659d6f83"}
Sep 08 13:40:59.654037 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:59.654009    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:40:59.676425 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:40:59.676365    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" podStartSLOduration=2.676349742 podStartE2EDuration="2.676349742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:40:59.674892106 +0000 UTC m=+1472.374822773" watchObservedRunningTime="2026-09-08 13:40:59.676349742 +0000 UTC m=+1472.376280376"
Sep 08 13:41:00.657015 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:00.656979    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:41:00.920098 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:00.920030    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:41:01.052111 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.052080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc8c230-e449-49d1-b580-5687be1198f6-proxy-tls\") pod \"4bc8c230-e449-49d1-b580-5687be1198f6\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") "
Sep 08 13:41:01.052285 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.052129    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt52n\" (UniqueName: \"kubernetes.io/projected/4bc8c230-e449-49d1-b580-5687be1198f6-kube-api-access-vt52n\") pod \"4bc8c230-e449-49d1-b580-5687be1198f6\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") "
Sep 08 13:41:01.052285 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.052155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc8c230-e449-49d1-b580-5687be1198f6-kube-rbac-proxy-sar-config\") pod \"4bc8c230-e449-49d1-b580-5687be1198f6\" (UID: \"4bc8c230-e449-49d1-b580-5687be1198f6\") "
Sep 08 13:41:01.052630 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.052600    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc8c230-e449-49d1-b580-5687be1198f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4bc8c230-e449-49d1-b580-5687be1198f6" (UID: "4bc8c230-e449-49d1-b580-5687be1198f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:41:01.054320 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.054290    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc8c230-e449-49d1-b580-5687be1198f6-kube-api-access-vt52n" (OuterVolumeSpecName: "kube-api-access-vt52n") pod "4bc8c230-e449-49d1-b580-5687be1198f6" (UID: "4bc8c230-e449-49d1-b580-5687be1198f6"). InnerVolumeSpecName "kube-api-access-vt52n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:41:01.054443 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.054418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc8c230-e449-49d1-b580-5687be1198f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bc8c230-e449-49d1-b580-5687be1198f6" (UID: "4bc8c230-e449-49d1-b580-5687be1198f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:41:01.153294 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.153262    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bc8c230-e449-49d1-b580-5687be1198f6-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:41:01.153294 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.153288    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt52n\" (UniqueName: \"kubernetes.io/projected/4bc8c230-e449-49d1-b580-5687be1198f6-kube-api-access-vt52n\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:41:01.153294 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.153300    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bc8c230-e449-49d1-b580-5687be1198f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:41:01.662210 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.662174    2580 generic.go:356] "Generic (PLEG): container finished" podID="4bc8c230-e449-49d1-b580-5687be1198f6" containerID="19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf" exitCode=0
Sep 08 13:41:01.662619 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.662223    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"
Sep 08 13:41:01.662619 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.662264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" event={"ID":"4bc8c230-e449-49d1-b580-5687be1198f6","Type":"ContainerDied","Data":"19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf"}
Sep 08 13:41:01.662619 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.662321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2" event={"ID":"4bc8c230-e449-49d1-b580-5687be1198f6","Type":"ContainerDied","Data":"80f7fef3eff1310d385b3f3f693dde2dcfdab290f0bdb95f36a9aace69fd743b"}
Sep 08 13:41:01.662619 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.662350    2580 scope.go:117] "RemoveContainer" containerID="de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89"
Sep 08 13:41:01.671536 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.671520    2580 scope.go:117] "RemoveContainer" containerID="19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf"
Sep 08 13:41:01.679283 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.679266    2580 scope.go:117] "RemoveContainer" containerID="de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89"
Sep 08 13:41:01.679591 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:41:01.679575    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89\": container with ID starting with de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89 not found: ID does not exist" containerID="de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89"
Sep 08 13:41:01.679656 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.679598    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89"} err="failed to get container status \"de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89\": rpc error: code = NotFound desc = could not find container \"de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89\": container with ID starting with de415547c5ca2a1f844d581e6739df88577bbb7de79061367f43bd48dc0eef89 not found: ID does not exist"
Sep 08 13:41:01.679656 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.679614    2580 scope.go:117] "RemoveContainer" containerID="19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf"
Sep 08 13:41:01.679815 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:41:01.679800    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf\": container with ID starting with 19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf not found: ID does not exist" containerID="19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf"
Sep 08 13:41:01.679854 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.679819    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf"} err="failed to get container status \"19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf\": rpc error: code = NotFound desc = could not find container \"19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf\": container with ID starting with 19690871989ba63e54b2c50bb3c7d0da9dfb885da90cb6062234f16b2b78ebaf not found: ID does not exist"
Sep 08 13:41:01.689350 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.689329    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"]
Sep 08 13:41:01.694997 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.694976    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-601fd-predictor-646cfc64df-ckjl2"]
Sep 08 13:41:01.751045 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:01.751010    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" path="/var/lib/kubelet/pods/4bc8c230-e449-49d1-b580-5687be1198f6/volumes"
Sep 08 13:41:02.527132 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:02.527090    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 08 13:41:05.662540 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:05.662508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:41:05.663143 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:05.663106    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:41:12.527540 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:12.527507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:41:15.664187 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:15.664144    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:41:22.920143 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:22.920103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:23.218388 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:23.218310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:25.664155 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:25.664111    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:41:33.037983 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.037950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:33.158428 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.158383    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"]
Sep 08 13:41:33.158744 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.158702    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container" containerID="cri-o://08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490" gracePeriod=30
Sep 08 13:41:33.158848 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.158741    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kube-rbac-proxy" containerID="cri-o://f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016" gracePeriod=30
Sep 08 13:41:33.187255 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.187219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:33.309751 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.309660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:33.328406 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.328357    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"]
Sep 08 13:41:33.328761 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.328744    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kube-rbac-proxy"
Sep 08 13:41:33.328844 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.328764    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kube-rbac-proxy"
Sep 08 13:41:33.328844 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.328781    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container"
Sep 08 13:41:33.328844 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.328790    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container"
Sep 08 13:41:33.329012 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.328888    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kube-rbac-proxy"
Sep 08 13:41:33.329012 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.328906    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bc8c230-e449-49d1-b580-5687be1198f6" containerName="kserve-container"
Sep 08 13:41:33.331970 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.331948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.334768 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.334739    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c7da0-kube-rbac-proxy-sar-config\""
Sep 08 13:41:33.335090 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.335074    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c7da0-predictor-serving-cert\""
Sep 08 13:41:33.347743 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.347719    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"]
Sep 08 13:41:33.427436 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.427385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0f9568-8937-4421-a904-2714533d3982-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7da0-predictor-945f88485-8srks\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") " pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.427609 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.427454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0f9568-8937-4421-a904-2714533d3982-proxy-tls\") pod \"success-200-isvc-c7da0-predictor-945f88485-8srks\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") " pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.427609 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.427508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd9tm\" (UniqueName: \"kubernetes.io/projected/4f0f9568-8937-4421-a904-2714533d3982-kube-api-access-xd9tm\") pod \"success-200-isvc-c7da0-predictor-945f88485-8srks\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") " pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.455841 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.455806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:33.528176 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.528137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd9tm\" (UniqueName: \"kubernetes.io/projected/4f0f9568-8937-4421-a904-2714533d3982-kube-api-access-xd9tm\") pod \"success-200-isvc-c7da0-predictor-945f88485-8srks\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") " pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.528176 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.528184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0f9568-8937-4421-a904-2714533d3982-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7da0-predictor-945f88485-8srks\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") " pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.528488 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.528219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0f9568-8937-4421-a904-2714533d3982-proxy-tls\") pod \"success-200-isvc-c7da0-predictor-945f88485-8srks\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") " pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.528931 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.528905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0f9568-8937-4421-a904-2714533d3982-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7da0-predictor-945f88485-8srks\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") " pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.530718 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.530696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0f9568-8937-4421-a904-2714533d3982-proxy-tls\") pod \"success-200-isvc-c7da0-predictor-945f88485-8srks\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") " pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.542028 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.542004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd9tm\" (UniqueName: \"kubernetes.io/projected/4f0f9568-8937-4421-a904-2714533d3982-kube-api-access-xd9tm\") pod \"success-200-isvc-c7da0-predictor-945f88485-8srks\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") " pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.601216 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.601181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:33.641646 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.641613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:33.807914 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.807885    2580 generic.go:356] "Generic (PLEG): container finished" podID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerID="f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016" exitCode=2
Sep 08 13:41:33.808073 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.807932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" event={"ID":"e5d5a9db-31ea-4c49-925e-5910db3e30da","Type":"ContainerDied","Data":"f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016"}
Sep 08 13:41:33.921743 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:33.921635    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"]
Sep 08 13:41:33.924755 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:41:33.924729    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0f9568_8937_4421_a904_2714533d3982.slice/crio-2309239abefeb5e4cf62bedb67de1cb2c5c74d1b5391ddc064a5caeb35f65863 WatchSource:0}: Error finding container 2309239abefeb5e4cf62bedb67de1cb2c5c74d1b5391ddc064a5caeb35f65863: Status 404 returned error can't find the container with id 2309239abefeb5e4cf62bedb67de1cb2c5c74d1b5391ddc064a5caeb35f65863
Sep 08 13:41:34.032020 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:34.031993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:34.138307 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:34.138275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:34.815337 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:34.815304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" event={"ID":"4f0f9568-8937-4421-a904-2714533d3982","Type":"ContainerStarted","Data":"6571ed1b7fd8321c3a8051d988a2442f598b6c312cb52d5071579c055fb42b19"}
Sep 08 13:41:34.815560 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:34.815366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:34.815560 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:34.815412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" event={"ID":"4f0f9568-8937-4421-a904-2714533d3982","Type":"ContainerStarted","Data":"2ba3fd3588426b2040374d0d0d48b506e32e9652a9c82de06eab1eefd6be7cb8"}
Sep 08 13:41:34.815560 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:34.815523    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:34.815710 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:34.815574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" event={"ID":"4f0f9568-8937-4421-a904-2714533d3982","Type":"ContainerStarted","Data":"2309239abefeb5e4cf62bedb67de1cb2c5c74d1b5391ddc064a5caeb35f65863"}
Sep 08 13:41:34.816815 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:34.816787    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:41:34.838989 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:34.838943    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podStartSLOduration=1.838927821 podStartE2EDuration="1.838927821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:34.837274968 +0000 UTC m=+1507.537205642" watchObservedRunningTime="2026-09-08 13:41:34.838927821 +0000 UTC m=+1507.538858455"
Sep 08 13:41:35.663466 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:35.663429    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:41:35.819426 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:35.819379    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:41:36.498667 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.498641    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:41:36.549640 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.549605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d5a9db-31ea-4c49-925e-5910db3e30da-proxy-tls\") pod \"e5d5a9db-31ea-4c49-925e-5910db3e30da\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") "
Sep 08 13:41:36.549809 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.549666    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2shs\" (UniqueName: \"kubernetes.io/projected/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-api-access-h2shs\") pod \"e5d5a9db-31ea-4c49-925e-5910db3e30da\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") "
Sep 08 13:41:36.549809 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.549689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-rbac-proxy-sar-config\") pod \"e5d5a9db-31ea-4c49-925e-5910db3e30da\" (UID: \"e5d5a9db-31ea-4c49-925e-5910db3e30da\") "
Sep 08 13:41:36.550118 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.550093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5d5a9db-31ea-4c49-925e-5910db3e30da" (UID: "e5d5a9db-31ea-4c49-925e-5910db3e30da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:41:36.551607 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.551580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d5a9db-31ea-4c49-925e-5910db3e30da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5d5a9db-31ea-4c49-925e-5910db3e30da" (UID: "e5d5a9db-31ea-4c49-925e-5910db3e30da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:41:36.551736 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.551719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-api-access-h2shs" (OuterVolumeSpecName: "kube-api-access-h2shs") pod "e5d5a9db-31ea-4c49-925e-5910db3e30da" (UID: "e5d5a9db-31ea-4c49-925e-5910db3e30da"). InnerVolumeSpecName "kube-api-access-h2shs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:41:36.650601 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.650573    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5d5a9db-31ea-4c49-925e-5910db3e30da-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:41:36.650601 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.650598    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2shs\" (UniqueName: \"kubernetes.io/projected/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-api-access-h2shs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:41:36.650782 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.650608    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5d5a9db-31ea-4c49-925e-5910db3e30da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:41:36.824259 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.824225    2580 generic.go:356] "Generic (PLEG): container finished" podID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerID="08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490" exitCode=0
Sep 08 13:41:36.824661 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.824275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"
Sep 08 13:41:36.824661 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.824313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" event={"ID":"e5d5a9db-31ea-4c49-925e-5910db3e30da","Type":"ContainerDied","Data":"08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490"}
Sep 08 13:41:36.824661 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.824376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z" event={"ID":"e5d5a9db-31ea-4c49-925e-5910db3e30da","Type":"ContainerDied","Data":"3b4ddbbd94db5eee2cf184a3906c06310bfeccf65f93807af2eb25b461008a9a"}
Sep 08 13:41:36.824661 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.824429    2580 scope.go:117] "RemoveContainer" containerID="f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016"
Sep 08 13:41:36.832865 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.832848    2580 scope.go:117] "RemoveContainer" containerID="08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490"
Sep 08 13:41:36.840246 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.840231    2580 scope.go:117] "RemoveContainer" containerID="f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016"
Sep 08 13:41:36.840507 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:41:36.840489    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016\": container with ID starting with f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016 not found: ID does not exist" containerID="f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016"
Sep 08 13:41:36.840585 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.840518    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016"} err="failed to get container status \"f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016\": rpc error: code = NotFound desc = could not find container \"f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016\": container with ID starting with f5747dc5720db8a133018bfaf7978e84b5e581c87addbfe20a333b5acccd2016 not found: ID does not exist"
Sep 08 13:41:36.840585 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.840539    2580 scope.go:117] "RemoveContainer" containerID="08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490"
Sep 08 13:41:36.840791 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:41:36.840775    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490\": container with ID starting with 08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490 not found: ID does not exist" containerID="08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490"
Sep 08 13:41:36.840841 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.840798    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490"} err="failed to get container status \"08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490\": rpc error: code = NotFound desc = could not find container \"08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490\": container with ID starting with 08710a67a1a5c00859aaf546f6c4538519055a65e9fa2bbeb57d2bb85327f490 not found: ID does not exist"
Sep 08 13:41:36.846865 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.846844    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"]
Sep 08 13:41:36.851291 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:36.851272    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e90df-predictor-5bbcb468bc-wkj7z"]
Sep 08 13:41:37.750511 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:37.750471    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" path="/var/lib/kubelet/pods/e5d5a9db-31ea-4c49-925e-5910db3e30da/volumes"
Sep 08 13:41:40.823985 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:40.823955    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:41:40.824419 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:40.824381    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:41:45.664560 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:45.664529    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:41:50.825183 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:50.825097    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:41:57.499337 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:57.499302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:58.134379 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:41:58.134348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:00.825232 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:42:00.825193    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:42:10.825266 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:42:10.825223    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:42:20.825544 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:42:20.825513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:42:33.270617 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:42:33.270579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:33.783257 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:42:33.783219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:50.476857 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:46:50.476825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:51.919531 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:48:51.919488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:12.178090 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.178056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:12.324523 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.324493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"]
Sep 08 13:50:12.324746 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.324719    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container" containerID="cri-o://d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222" gracePeriod=30
Sep 08 13:50:12.324823 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.324759    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kube-rbac-proxy" containerID="cri-o://ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff" gracePeriod=30
Sep 08 13:50:12.354193 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.354154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:12.416662 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.416624    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"]
Sep 08 13:50:12.417047 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.417031    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container"
Sep 08 13:50:12.417128 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.417049    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container"
Sep 08 13:50:12.417128 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.417058    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kube-rbac-proxy"
Sep 08 13:50:12.417128 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.417064    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kube-rbac-proxy"
Sep 08 13:50:12.417128 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.417119    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kserve-container"
Sep 08 13:50:12.417128 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.417126    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d5a9db-31ea-4c49-925e-5910db3e30da" containerName="kube-rbac-proxy"
Sep 08 13:50:12.420168 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.420153    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.422487 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.422456    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-13b40-predictor-serving-cert\""
Sep 08 13:50:12.422875 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.422857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-13b40-kube-rbac-proxy-sar-config\""
Sep 08 13:50:12.436604 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.436574    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"]
Sep 08 13:50:12.452743 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.452716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:12.522891 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.522857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9nq\" (UniqueName: \"kubernetes.io/projected/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-api-access-fv9nq\") pod \"success-200-isvc-13b40-predictor-8bb687fb7-g48nb\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") " pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.523070 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.522909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-13b40-predictor-8bb687fb7-g48nb\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") " pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.523070 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.522986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1655960d-4ef9-47b9-97de-c266d9e1d18e-proxy-tls\") pod \"success-200-isvc-13b40-predictor-8bb687fb7-g48nb\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") " pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.624088 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.624052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9nq\" (UniqueName: \"kubernetes.io/projected/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-api-access-fv9nq\") pod \"success-200-isvc-13b40-predictor-8bb687fb7-g48nb\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") " pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.624296 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.624099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-13b40-predictor-8bb687fb7-g48nb\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") " pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.624296 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.624161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1655960d-4ef9-47b9-97de-c266d9e1d18e-proxy-tls\") pod \"success-200-isvc-13b40-predictor-8bb687fb7-g48nb\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") " pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.624920 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.624895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-13b40-predictor-8bb687fb7-g48nb\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") " pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.626547 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.626526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1655960d-4ef9-47b9-97de-c266d9e1d18e-proxy-tls\") pod \"success-200-isvc-13b40-predictor-8bb687fb7-g48nb\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") " pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.633097 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.633077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9nq\" (UniqueName: \"kubernetes.io/projected/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-api-access-fv9nq\") pod \"success-200-isvc-13b40-predictor-8bb687fb7-g48nb\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") " pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.733943 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.733872    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:12.908726 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.908673    2580 generic.go:356] "Generic (PLEG): container finished" podID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerID="ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff" exitCode=2
Sep 08 13:50:12.908893 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:12.908743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" event={"ID":"68c6a880-7d55-497a-8b8e-f3f62c7562cd","Type":"ContainerDied","Data":"ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff"}
Sep 08 13:50:13.010355 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.010279    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"]
Sep 08 13:50:13.013905 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:50:13.013881    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1655960d_4ef9_47b9_97de_c266d9e1d18e.slice/crio-f8cd5bb5551a7eebd1f3bf126a4db95829ee0b2167bf4b6741c386a32f6f3d5a WatchSource:0}: Error finding container f8cd5bb5551a7eebd1f3bf126a4db95829ee0b2167bf4b6741c386a32f6f3d5a: Status 404 returned error can't find the container with id f8cd5bb5551a7eebd1f3bf126a4db95829ee0b2167bf4b6741c386a32f6f3d5a
Sep 08 13:50:13.044191 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.044164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:13.115231 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.115204    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:13.920739 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.920687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" event={"ID":"1655960d-4ef9-47b9-97de-c266d9e1d18e","Type":"ContainerStarted","Data":"8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383"}
Sep 08 13:50:13.921171 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.920795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" event={"ID":"1655960d-4ef9-47b9-97de-c266d9e1d18e","Type":"ContainerStarted","Data":"977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761"}
Sep 08 13:50:13.921171 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.920826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" event={"ID":"1655960d-4ef9-47b9-97de-c266d9e1d18e","Type":"ContainerStarted","Data":"f8cd5bb5551a7eebd1f3bf126a4db95829ee0b2167bf4b6741c386a32f6f3d5a"}
Sep 08 13:50:13.921171 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.920984    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:13.921171 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.921088    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:13.922473 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.922448    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:50:13.944580 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:13.944532    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podStartSLOduration=1.9445182989999998 podStartE2EDuration="1.944518299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:13.942967639 +0000 UTC m=+2026.642898498" watchObservedRunningTime="2026-09-08 13:50:13.944518299 +0000 UTC m=+2026.644448936"
Sep 08 13:50:14.925390 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:14.925353    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:50:15.568778 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.568753    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:50:15.650238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.650205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-rbac-proxy-sar-config\") pod \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") "
Sep 08 13:50:15.650238 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.650243    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c6a880-7d55-497a-8b8e-f3f62c7562cd-proxy-tls\") pod \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") "
Sep 08 13:50:15.650467 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.650328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v7bf\" (UniqueName: \"kubernetes.io/projected/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-api-access-5v7bf\") pod \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\" (UID: \"68c6a880-7d55-497a-8b8e-f3f62c7562cd\") "
Sep 08 13:50:15.650579 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.650554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "68c6a880-7d55-497a-8b8e-f3f62c7562cd" (UID: "68c6a880-7d55-497a-8b8e-f3f62c7562cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:50:15.652381 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.652344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-api-access-5v7bf" (OuterVolumeSpecName: "kube-api-access-5v7bf") pod "68c6a880-7d55-497a-8b8e-f3f62c7562cd" (UID: "68c6a880-7d55-497a-8b8e-f3f62c7562cd"). InnerVolumeSpecName "kube-api-access-5v7bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:50:15.652381 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.652357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c6a880-7d55-497a-8b8e-f3f62c7562cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68c6a880-7d55-497a-8b8e-f3f62c7562cd" (UID: "68c6a880-7d55-497a-8b8e-f3f62c7562cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:50:15.750975 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.750946    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v7bf\" (UniqueName: \"kubernetes.io/projected/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-api-access-5v7bf\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:50:15.750975 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.750968    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c6a880-7d55-497a-8b8e-f3f62c7562cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:50:15.750975 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.750979    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c6a880-7d55-497a-8b8e-f3f62c7562cd-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:50:15.931013 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.930927    2580 generic.go:356] "Generic (PLEG): container finished" podID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerID="d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222" exitCode=0
Sep 08 13:50:15.931013 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.930977    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"
Sep 08 13:50:15.931013 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.930992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" event={"ID":"68c6a880-7d55-497a-8b8e-f3f62c7562cd","Type":"ContainerDied","Data":"d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222"}
Sep 08 13:50:15.931576 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.931053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r" event={"ID":"68c6a880-7d55-497a-8b8e-f3f62c7562cd","Type":"ContainerDied","Data":"6e8f15e911dfee2221708fa8ed31da78b9d2863493c4808de6cbe600659d6f83"}
Sep 08 13:50:15.931576 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.931090    2580 scope.go:117] "RemoveContainer" containerID="ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff"
Sep 08 13:50:15.939232 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.939213    2580 scope.go:117] "RemoveContainer" containerID="d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222"
Sep 08 13:50:15.946558 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.946532    2580 scope.go:117] "RemoveContainer" containerID="ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff"
Sep 08 13:50:15.946821 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:50:15.946800    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff\": container with ID starting with ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff not found: ID does not exist" containerID="ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff"
Sep 08 13:50:15.946893 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.946833    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff"} err="failed to get container status \"ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff\": rpc error: code = NotFound desc = could not find container \"ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff\": container with ID starting with ee4b996bee9ab15270cf1e7465005c504f3d0dca78e15cc2d05dc0081f42f0ff not found: ID does not exist"
Sep 08 13:50:15.946893 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.946854    2580 scope.go:117] "RemoveContainer" containerID="d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222"
Sep 08 13:50:15.947134 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:50:15.947118    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222\": container with ID starting with d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222 not found: ID does not exist" containerID="d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222"
Sep 08 13:50:15.947170 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.947140    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222"} err="failed to get container status \"d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222\": rpc error: code = NotFound desc = could not find container \"d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222\": container with ID starting with d53d06d18ddc61d862987cf652ef7572ecb05d8d1bb672dba4c45a2f7bc75222 not found: ID does not exist"
Sep 08 13:50:15.957597 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.957575    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"]
Sep 08 13:50:15.962187 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:15.962166    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64898-predictor-57884f876f-8wq7r"]
Sep 08 13:50:17.750655 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:17.750621    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" path="/var/lib/kubelet/pods/68c6a880-7d55-497a-8b8e-f3f62c7562cd/volumes"
Sep 08 13:50:19.929900 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:19.929865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:50:19.930377 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:19.930350    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:50:20.761621 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:20.761590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:20.840905 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:20.840868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:29.930306 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:29.930264    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:50:39.931051 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:39.931005    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:50:47.745296 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.745255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:47.833438 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.833405    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"]
Sep 08 13:50:47.833703 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.833650    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container" containerID="cri-o://2ba3fd3588426b2040374d0d0d48b506e32e9652a9c82de06eab1eefd6be7cb8" gracePeriod=30
Sep 08 13:50:47.833775 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.833691    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kube-rbac-proxy" containerID="cri-o://6571ed1b7fd8321c3a8051d988a2442f598b6c312cb52d5071579c055fb42b19" gracePeriod=30
Sep 08 13:50:47.869298 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.869264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:47.927922 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.927892    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"]
Sep 08 13:50:47.928274 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.928260    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container"
Sep 08 13:50:47.928345 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.928277    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container"
Sep 08 13:50:47.928345 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.928310    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kube-rbac-proxy"
Sep 08 13:50:47.928345 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.928318    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kube-rbac-proxy"
Sep 08 13:50:47.928485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.928428    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kserve-container"
Sep 08 13:50:47.928485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.928443    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c6a880-7d55-497a-8b8e-f3f62c7562cd" containerName="kube-rbac-proxy"
Sep 08 13:50:47.931438 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.931381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:47.935032 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.935008    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ce1f6-predictor-serving-cert\""
Sep 08 13:50:47.935347 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.935329    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ce1f6-kube-rbac-proxy-sar-config\""
Sep 08 13:50:47.943497 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.943463    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"]
Sep 08 13:50:47.969295 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:47.969265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:48.020229 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.020137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6077a8b1-0ed9-4e61-ba96-09329e8af889-proxy-tls\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.020229 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.020204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxcdj\" (UniqueName: \"kubernetes.io/projected/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-api-access-lxcdj\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.020487 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.020282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.063731 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.063701    2580 generic.go:356] "Generic (PLEG): container finished" podID="4f0f9568-8937-4421-a904-2714533d3982" containerID="6571ed1b7fd8321c3a8051d988a2442f598b6c312cb52d5071579c055fb42b19" exitCode=2
Sep 08 13:50:48.063916 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.063776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" event={"ID":"4f0f9568-8937-4421-a904-2714533d3982","Type":"ContainerDied","Data":"6571ed1b7fd8321c3a8051d988a2442f598b6c312cb52d5071579c055fb42b19"}
Sep 08 13:50:48.073072 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.073043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:48.120803 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.120773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6077a8b1-0ed9-4e61-ba96-09329e8af889-proxy-tls\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.121011 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.120818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcdj\" (UniqueName: \"kubernetes.io/projected/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-api-access-lxcdj\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.121011 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.120879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.121011 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:50:48.120946    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-serving-cert: secret "success-200-isvc-ce1f6-predictor-serving-cert" not found
Sep 08 13:50:48.121187 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:50:48.121033    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6077a8b1-0ed9-4e61-ba96-09329e8af889-proxy-tls podName:6077a8b1-0ed9-4e61-ba96-09329e8af889 nodeName:}" failed. No retries permitted until 2026-09-08 13:50:48.621010429 +0000 UTC m=+2061.320941056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6077a8b1-0ed9-4e61-ba96-09329e8af889-proxy-tls") pod "success-200-isvc-ce1f6-predictor-76644689b9-smjt7" (UID: "6077a8b1-0ed9-4e61-ba96-09329e8af889") : secret "success-200-isvc-ce1f6-predictor-serving-cert" not found
Sep 08 13:50:48.121704 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.121657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.135079 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.135057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxcdj\" (UniqueName: \"kubernetes.io/projected/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-api-access-lxcdj\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.162737 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.162701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:48.274977 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.274900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:48.625085 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.625049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6077a8b1-0ed9-4e61-ba96-09329e8af889-proxy-tls\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.627607 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.627584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6077a8b1-0ed9-4e61-ba96-09329e8af889-proxy-tls\") pod \"success-200-isvc-ce1f6-predictor-76644689b9-smjt7\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") " pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:48.670862 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.670825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:48.844886 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:48.844857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:49.124840 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:49.124806    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"]
Sep 08 13:50:49.127883 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:50:49.127856    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6077a8b1_0ed9_4e61_ba96_09329e8af889.slice/crio-66bc59d8dc44c2106ddf602904d43b39f84cdec4a2d9e76ad9d7f75dc67eb59c WatchSource:0}: Error finding container 66bc59d8dc44c2106ddf602904d43b39f84cdec4a2d9e76ad9d7f75dc67eb59c: Status 404 returned error can't find the container with id 66bc59d8dc44c2106ddf602904d43b39f84cdec4a2d9e76ad9d7f75dc67eb59c
Sep 08 13:50:49.930879 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:49.930843    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:50:50.074573 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:50.074497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" event={"ID":"6077a8b1-0ed9-4e61-ba96-09329e8af889","Type":"ContainerStarted","Data":"f29748f1ed91757750d2a6e63b947e20a6a37a5b89f57166aee13c1d7bbf3bca"}
Sep 08 13:50:50.074573 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:50.074554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" event={"ID":"6077a8b1-0ed9-4e61-ba96-09329e8af889","Type":"ContainerStarted","Data":"399ef8bda34ac40ea0e1bdb64c1e1fabdaff6fdbdb7a299c1d8c0ed608e3da3f"}
Sep 08 13:50:50.074778 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:50.074583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" event={"ID":"6077a8b1-0ed9-4e61-ba96-09329e8af889","Type":"ContainerStarted","Data":"66bc59d8dc44c2106ddf602904d43b39f84cdec4a2d9e76ad9d7f75dc67eb59c"}
Sep 08 13:50:50.074778 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:50.074667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:50.112242 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:50.112188    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podStartSLOduration=3.112172127 podStartE2EDuration="3.112172127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:50.111271651 +0000 UTC m=+2062.811202288" watchObservedRunningTime="2026-09-08 13:50:50.112172127 +0000 UTC m=+2062.812102760"
Sep 08 13:50:50.820492 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:50.820448    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 08 13:50:50.824733 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:50.824704    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:50:51.081135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.081067    2580 generic.go:356] "Generic (PLEG): container finished" podID="4f0f9568-8937-4421-a904-2714533d3982" containerID="2ba3fd3588426b2040374d0d0d48b506e32e9652a9c82de06eab1eefd6be7cb8" exitCode=0
Sep 08 13:50:51.081472 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.081132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" event={"ID":"4f0f9568-8937-4421-a904-2714533d3982","Type":"ContainerDied","Data":"2ba3fd3588426b2040374d0d0d48b506e32e9652a9c82de06eab1eefd6be7cb8"}
Sep 08 13:50:51.081472 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.081446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:51.082753 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.082729    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:50:51.177729 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.177704    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:50:51.351146 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.351109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd9tm\" (UniqueName: \"kubernetes.io/projected/4f0f9568-8937-4421-a904-2714533d3982-kube-api-access-xd9tm\") pod \"4f0f9568-8937-4421-a904-2714533d3982\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") "
Sep 08 13:50:51.351146 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.351150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0f9568-8937-4421-a904-2714533d3982-kube-rbac-proxy-sar-config\") pod \"4f0f9568-8937-4421-a904-2714533d3982\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") "
Sep 08 13:50:51.351362 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.351171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0f9568-8937-4421-a904-2714533d3982-proxy-tls\") pod \"4f0f9568-8937-4421-a904-2714533d3982\" (UID: \"4f0f9568-8937-4421-a904-2714533d3982\") "
Sep 08 13:50:51.351550 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.351527    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0f9568-8937-4421-a904-2714533d3982-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f0f9568-8937-4421-a904-2714533d3982" (UID: "4f0f9568-8937-4421-a904-2714533d3982"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:50:51.353324 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.353300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0f9568-8937-4421-a904-2714533d3982-kube-api-access-xd9tm" (OuterVolumeSpecName: "kube-api-access-xd9tm") pod "4f0f9568-8937-4421-a904-2714533d3982" (UID: "4f0f9568-8937-4421-a904-2714533d3982"). InnerVolumeSpecName "kube-api-access-xd9tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:50:51.353451 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.353338    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0f9568-8937-4421-a904-2714533d3982-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f0f9568-8937-4421-a904-2714533d3982" (UID: "4f0f9568-8937-4421-a904-2714533d3982"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:50:51.452796 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.452758    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd9tm\" (UniqueName: \"kubernetes.io/projected/4f0f9568-8937-4421-a904-2714533d3982-kube-api-access-xd9tm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:50:51.452796 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.452789    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f0f9568-8937-4421-a904-2714533d3982-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:50:51.452796 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:51.452800    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f0f9568-8937-4421-a904-2714533d3982-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:50:52.086801 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:52.086766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks" event={"ID":"4f0f9568-8937-4421-a904-2714533d3982","Type":"ContainerDied","Data":"2309239abefeb5e4cf62bedb67de1cb2c5c74d1b5391ddc064a5caeb35f65863"}
Sep 08 13:50:52.086801 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:52.086800    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"
Sep 08 13:50:52.087300 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:52.086853    2580 scope.go:117] "RemoveContainer" containerID="6571ed1b7fd8321c3a8051d988a2442f598b6c312cb52d5071579c055fb42b19"
Sep 08 13:50:52.087300 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:52.087103    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:50:52.094890 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:52.094870    2580 scope.go:117] "RemoveContainer" containerID="2ba3fd3588426b2040374d0d0d48b506e32e9652a9c82de06eab1eefd6be7cb8"
Sep 08 13:50:52.118278 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:52.118252    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"]
Sep 08 13:50:52.121255 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:52.121233    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7da0-predictor-945f88485-8srks"]
Sep 08 13:50:53.750357 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:53.750323    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f0f9568-8937-4421-a904-2714533d3982" path="/var/lib/kubelet/pods/4f0f9568-8937-4421-a904-2714533d3982/volumes"
Sep 08 13:50:57.091365 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:57.091337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:50:57.091875 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:57.091850    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:50:59.931552 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:50:59.931521    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:51:07.092815 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:07.092766    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:51:12.469337 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:12.469301    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:12.786612 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:12.786526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:13.422474 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:13.422428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:17.092632 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:17.092593    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:51:22.547234 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.547151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:22.633348 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.633317    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"]
Sep 08 13:51:22.633574 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.633549    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container" containerID="cri-o://977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761" gracePeriod=30
Sep 08 13:51:22.633631 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.633581    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kube-rbac-proxy" containerID="cri-o://8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383" gracePeriod=30
Sep 08 13:51:22.726861 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.726821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:22.824279 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.824194    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"]
Sep 08 13:51:22.824555 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.824542    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kube-rbac-proxy"
Sep 08 13:51:22.824625 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.824557    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kube-rbac-proxy"
Sep 08 13:51:22.824625 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.824569    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container"
Sep 08 13:51:22.824625 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.824577    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container"
Sep 08 13:51:22.824776 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.824658    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kube-rbac-proxy"
Sep 08 13:51:22.824776 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.824669    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0f9568-8937-4421-a904-2714533d3982" containerName="kserve-container"
Sep 08 13:51:22.827705 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.827688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:22.830098 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.830081    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7d014-predictor-serving-cert\""
Sep 08 13:51:22.830234 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.830084    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7d014-kube-rbac-proxy-sar-config\""
Sep 08 13:51:22.842998 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.842972    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"]
Sep 08 13:51:22.863707 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.863672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:22.916061 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.916028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb32918-a428-4bc0-a5bb-c22184f68e08-proxy-tls\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:22.916228 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.916070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwstm\" (UniqueName: \"kubernetes.io/projected/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-api-access-fwstm\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:22.916228 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.916092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:22.942256 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:22.942222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:23.017732 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.017690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb32918-a428-4bc0-a5bb-c22184f68e08-proxy-tls\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:23.017924 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.017757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwstm\" (UniqueName: \"kubernetes.io/projected/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-api-access-fwstm\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:23.017924 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.017789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:23.017924 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:51:23.017862    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7d014-predictor-serving-cert: secret "success-200-isvc-7d014-predictor-serving-cert" not found
Sep 08 13:51:23.018168 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:51:23.017932    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb32918-a428-4bc0-a5bb-c22184f68e08-proxy-tls podName:6cb32918-a428-4bc0-a5bb-c22184f68e08 nodeName:}" failed. No retries permitted until 2026-09-08 13:51:23.51791296 +0000 UTC m=+2096.217843587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6cb32918-a428-4bc0-a5bb-c22184f68e08-proxy-tls") pod "success-200-isvc-7d014-predictor-5688f58dbc-7pbff" (UID: "6cb32918-a428-4bc0-a5bb-c22184f68e08") : secret "success-200-isvc-7d014-predictor-serving-cert" not found
Sep 08 13:51:23.018605 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.018584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:23.026223 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.026194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwstm\" (UniqueName: \"kubernetes.io/projected/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-api-access-fwstm\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:23.215334 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.215302    2580 generic.go:356] "Generic (PLEG): container finished" podID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerID="8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383" exitCode=2
Sep 08 13:51:23.215503 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.215372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" event={"ID":"1655960d-4ef9-47b9-97de-c266d9e1d18e","Type":"ContainerDied","Data":"8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383"}
Sep 08 13:51:23.267116 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.267081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:23.511681 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.511567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:23.523601 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.523573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb32918-a428-4bc0-a5bb-c22184f68e08-proxy-tls\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:23.526022 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.525999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb32918-a428-4bc0-a5bb-c22184f68e08-proxy-tls\") pod \"success-200-isvc-7d014-predictor-5688f58dbc-7pbff\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") " pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:23.738126 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:23.738091    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:24.009121 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:24.009089    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"]
Sep 08 13:51:24.011037 ip-10-0-140-239 kubenswrapper[2580]: W0908 13:51:24.011007    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb32918_a428_4bc0_a5bb_c22184f68e08.slice/crio-dbffdb8f90554b79c3368f2dabe53e771d153a0e5911735ad62b33b3ef44db53 WatchSource:0}: Error finding container dbffdb8f90554b79c3368f2dabe53e771d153a0e5911735ad62b33b3ef44db53: Status 404 returned error can't find the container with id dbffdb8f90554b79c3368f2dabe53e771d153a0e5911735ad62b33b3ef44db53
Sep 08 13:51:24.222287 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:24.222247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" event={"ID":"6cb32918-a428-4bc0-a5bb-c22184f68e08","Type":"ContainerStarted","Data":"74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0"}
Sep 08 13:51:24.222475 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:24.222310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" event={"ID":"6cb32918-a428-4bc0-a5bb-c22184f68e08","Type":"ContainerStarted","Data":"47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a"}
Sep 08 13:51:24.222475 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:24.222338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" event={"ID":"6cb32918-a428-4bc0-a5bb-c22184f68e08","Type":"ContainerStarted","Data":"dbffdb8f90554b79c3368f2dabe53e771d153a0e5911735ad62b33b3ef44db53"}
Sep 08 13:51:24.222475 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:24.222368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:24.248849 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:24.248789    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" podStartSLOduration=2.248772463 podStartE2EDuration="2.248772463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:51:24.247386892 +0000 UTC m=+2096.947317552" watchObservedRunningTime="2026-09-08 13:51:24.248772463 +0000 UTC m=+2096.948703097"
Sep 08 13:51:24.926104 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:24.926060    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 08 13:51:25.226864 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:25.226778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:25.228100 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:25.228068    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:51:25.879485 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:25.879452    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:51:25.945022 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:25.944982    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-rbac-proxy-sar-config\") pod \"1655960d-4ef9-47b9-97de-c266d9e1d18e\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") "
Sep 08 13:51:25.945483 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:25.945067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv9nq\" (UniqueName: \"kubernetes.io/projected/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-api-access-fv9nq\") pod \"1655960d-4ef9-47b9-97de-c266d9e1d18e\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") "
Sep 08 13:51:25.945483 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:25.945113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1655960d-4ef9-47b9-97de-c266d9e1d18e-proxy-tls\") pod \"1655960d-4ef9-47b9-97de-c266d9e1d18e\" (UID: \"1655960d-4ef9-47b9-97de-c266d9e1d18e\") "
Sep 08 13:51:25.945483 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:25.945366    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1655960d-4ef9-47b9-97de-c266d9e1d18e" (UID: "1655960d-4ef9-47b9-97de-c266d9e1d18e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:51:25.947252 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:25.947224    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-api-access-fv9nq" (OuterVolumeSpecName: "kube-api-access-fv9nq") pod "1655960d-4ef9-47b9-97de-c266d9e1d18e" (UID: "1655960d-4ef9-47b9-97de-c266d9e1d18e"). InnerVolumeSpecName "kube-api-access-fv9nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:51:25.947352 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:25.947223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1655960d-4ef9-47b9-97de-c266d9e1d18e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1655960d-4ef9-47b9-97de-c266d9e1d18e" (UID: "1655960d-4ef9-47b9-97de-c266d9e1d18e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:51:26.045956 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.045794    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:51:26.045956 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.045826    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv9nq\" (UniqueName: \"kubernetes.io/projected/1655960d-4ef9-47b9-97de-c266d9e1d18e-kube-api-access-fv9nq\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:51:26.045956 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.045839    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1655960d-4ef9-47b9-97de-c266d9e1d18e-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 13:51:26.232354 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.232319    2580 generic.go:356] "Generic (PLEG): container finished" podID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerID="977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761" exitCode=0
Sep 08 13:51:26.232545 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.232378    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"
Sep 08 13:51:26.232545 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.232416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" event={"ID":"1655960d-4ef9-47b9-97de-c266d9e1d18e","Type":"ContainerDied","Data":"977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761"}
Sep 08 13:51:26.232545 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.232489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb" event={"ID":"1655960d-4ef9-47b9-97de-c266d9e1d18e","Type":"ContainerDied","Data":"f8cd5bb5551a7eebd1f3bf126a4db95829ee0b2167bf4b6741c386a32f6f3d5a"}
Sep 08 13:51:26.232545 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.232536    2580 scope.go:117] "RemoveContainer" containerID="8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383"
Sep 08 13:51:26.232998 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.232962    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:51:26.241135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.241119    2580 scope.go:117] "RemoveContainer" containerID="977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761"
Sep 08 13:51:26.248515 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.248499    2580 scope.go:117] "RemoveContainer" containerID="8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383"
Sep 08 13:51:26.248780 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:51:26.248763    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383\": container with ID starting with 8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383 not found: ID does not exist" containerID="8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383"
Sep 08 13:51:26.248836 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.248786    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383"} err="failed to get container status \"8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383\": rpc error: code = NotFound desc = could not find container \"8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383\": container with ID starting with 8c13892ff141fe35ec41e3343c4ed6afc72024f7eceea13387eb4bee9792a383 not found: ID does not exist"
Sep 08 13:51:26.248836 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.248813    2580 scope.go:117] "RemoveContainer" containerID="977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761"
Sep 08 13:51:26.249082 ip-10-0-140-239 kubenswrapper[2580]: E0908 13:51:26.249066    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761\": container with ID starting with 977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761 not found: ID does not exist" containerID="977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761"
Sep 08 13:51:26.249135 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.249084    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761"} err="failed to get container status \"977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761\": rpc error: code = NotFound desc = could not find container \"977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761\": container with ID starting with 977579e89bd102c1982a24bdc02906c6576ade3f25977bfd2e51b9962b8d5761 not found: ID does not exist"
Sep 08 13:51:26.263697 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.263663    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"]
Sep 08 13:51:26.267673 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:26.267649    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13b40-predictor-8bb687fb7-g48nb"]
Sep 08 13:51:27.092762 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:27.092722    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:51:27.750362 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:27.750332    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" path="/var/lib/kubelet/pods/1655960d-4ef9-47b9-97de-c266d9e1d18e/volumes"
Sep 08 13:51:31.237791 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:31.237761    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:51:31.238250 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:31.238220    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:51:37.092580 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:37.092544    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 13:51:41.239039 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:41.239001    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:51:48.101163 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:48.101127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:48.196229 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:48.196197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:51.238290 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:51:51.238253    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:52:01.238994 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:52:01.238945    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:52:11.239154 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:52:11.239122    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 13:52:22.804726 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:52:22.804686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:24.526257 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:52:24.526224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:02.908375 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:56:02.908342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:20.910298 ip-10-0-140-239 kubenswrapper[2580]: I0908 13:59:20.910258    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:37.489303 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:37.489268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:37.558560 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:37.558531    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"]
Sep 08 14:00:37.558827 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:37.558800    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container" containerID="cri-o://47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a" gracePeriod=30
Sep 08 14:00:37.559002 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:37.558942    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kube-rbac-proxy" containerID="cri-o://74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0" gracePeriod=30
Sep 08 14:00:37.594692 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:37.594638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:37.680963 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:37.680934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:38.432936 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:38.432900    2580 generic.go:356] "Generic (PLEG): container finished" podID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerID="74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0" exitCode=2
Sep 08 14:00:38.433108 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:38.432945    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" event={"ID":"6cb32918-a428-4bc0-a5bb-c22184f68e08","Type":"ContainerDied","Data":"74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0"}
Sep 08 14:00:40.803327 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:40.803303    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 14:00:40.906716 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:40.906621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb32918-a428-4bc0-a5bb-c22184f68e08-proxy-tls\") pod \"6cb32918-a428-4bc0-a5bb-c22184f68e08\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") "
Sep 08 14:00:40.906716 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:40.906691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwstm\" (UniqueName: \"kubernetes.io/projected/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-api-access-fwstm\") pod \"6cb32918-a428-4bc0-a5bb-c22184f68e08\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") "
Sep 08 14:00:40.906940 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:40.906789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-rbac-proxy-sar-config\") pod \"6cb32918-a428-4bc0-a5bb-c22184f68e08\" (UID: \"6cb32918-a428-4bc0-a5bb-c22184f68e08\") "
Sep 08 14:00:40.907134 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:40.907106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6cb32918-a428-4bc0-a5bb-c22184f68e08" (UID: "6cb32918-a428-4bc0-a5bb-c22184f68e08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:40.908672 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:40.908643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-api-access-fwstm" (OuterVolumeSpecName: "kube-api-access-fwstm") pod "6cb32918-a428-4bc0-a5bb-c22184f68e08" (UID: "6cb32918-a428-4bc0-a5bb-c22184f68e08"). InnerVolumeSpecName "kube-api-access-fwstm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:40.908752 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:40.908683    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb32918-a428-4bc0-a5bb-c22184f68e08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6cb32918-a428-4bc0-a5bb-c22184f68e08" (UID: "6cb32918-a428-4bc0-a5bb-c22184f68e08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:00:41.007815 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.007782    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb32918-a428-4bc0-a5bb-c22184f68e08-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 14:00:41.007815 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.007811    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwstm\" (UniqueName: \"kubernetes.io/projected/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-api-access-fwstm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 14:00:41.007815 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.007821    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb32918-a428-4bc0-a5bb-c22184f68e08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 14:00:41.445934 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.445893    2580 generic.go:356] "Generic (PLEG): container finished" podID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerID="47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a" exitCode=0
Sep 08 14:00:41.446112 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.445949    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"
Sep 08 14:00:41.446112 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.445975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" event={"ID":"6cb32918-a428-4bc0-a5bb-c22184f68e08","Type":"ContainerDied","Data":"47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a"}
Sep 08 14:00:41.446112 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.446036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff" event={"ID":"6cb32918-a428-4bc0-a5bb-c22184f68e08","Type":"ContainerDied","Data":"dbffdb8f90554b79c3368f2dabe53e771d153a0e5911735ad62b33b3ef44db53"}
Sep 08 14:00:41.446112 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.446063    2580 scope.go:117] "RemoveContainer" containerID="74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0"
Sep 08 14:00:41.454790 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.454768    2580 scope.go:117] "RemoveContainer" containerID="47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a"
Sep 08 14:00:41.462430 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.462380    2580 scope.go:117] "RemoveContainer" containerID="74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0"
Sep 08 14:00:41.463158 ip-10-0-140-239 kubenswrapper[2580]: E0908 14:00:41.463137    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0\": container with ID starting with 74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0 not found: ID does not exist" containerID="74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0"
Sep 08 14:00:41.463236 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.463167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0"} err="failed to get container status \"74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0\": rpc error: code = NotFound desc = could not find container \"74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0\": container with ID starting with 74f29b1d4af918446aa1d1941376a36502618e1b58659aa63c36ebf9b45161a0 not found: ID does not exist"
Sep 08 14:00:41.463236 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.463185    2580 scope.go:117] "RemoveContainer" containerID="47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a"
Sep 08 14:00:41.463523 ip-10-0-140-239 kubenswrapper[2580]: E0908 14:00:41.463500    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a\": container with ID starting with 47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a not found: ID does not exist" containerID="47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a"
Sep 08 14:00:41.463575 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.463531    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a"} err="failed to get container status \"47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a\": rpc error: code = NotFound desc = could not find container \"47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a\": container with ID starting with 47c2f8b47cc5bad86b78da9d25109574287c5c0dfda141da3d78f8eb3a63927a not found: ID does not exist"
Sep 08 14:00:41.467502 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.467479    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"]
Sep 08 14:00:41.469445 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.469425    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7d014-predictor-5688f58dbc-7pbff"]
Sep 08 14:00:41.749843 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:00:41.749773    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" path="/var/lib/kubelet/pods/6cb32918-a428-4bc0-a5bb-c22184f68e08/volumes"
Sep 08 14:05:41.217549 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:05:41.217510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:07.139455 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:06:07.139391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:07.386758 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:07.386727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:07.474910 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:07.474871    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"]
Sep 08 14:08:07.475180 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:07.475138    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container" containerID="cri-o://399ef8bda34ac40ea0e1bdb64c1e1fabdaff6fdbdb7a299c1d8c0ed608e3da3f" gracePeriod=30
Sep 08 14:08:07.475301 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:07.475176    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kube-rbac-proxy" containerID="cri-o://f29748f1ed91757750d2a6e63b947e20a6a37a5b89f57166aee13c1d7bbf3bca" gracePeriod=30
Sep 08 14:08:07.503604 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:07.503575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:07.571436 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:07.571373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:08.203522 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.203491    2580 generic.go:356] "Generic (PLEG): container finished" podID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerID="f29748f1ed91757750d2a6e63b947e20a6a37a5b89f57166aee13c1d7bbf3bca" exitCode=2
Sep 08 14:08:08.203706 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.203567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" event={"ID":"6077a8b1-0ed9-4e61-ba96-09329e8af889","Type":"ContainerDied","Data":"f29748f1ed91757750d2a6e63b947e20a6a37a5b89f57166aee13c1d7bbf3bca"}
Sep 08 14:08:08.735362 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735334    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2qsb2/must-gather-5rt2n"]
Sep 08 14:08:08.735761 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735647    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kube-rbac-proxy"
Sep 08 14:08:08.735761 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735658    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kube-rbac-proxy"
Sep 08 14:08:08.735761 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735673    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container"
Sep 08 14:08:08.735761 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735679    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container"
Sep 08 14:08:08.735761 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735689    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kube-rbac-proxy"
Sep 08 14:08:08.735761 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735694    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kube-rbac-proxy"
Sep 08 14:08:08.735761 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735711    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container"
Sep 08 14:08:08.735761 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735716    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container"
Sep 08 14:08:08.735761 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735765    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kserve-container"
Sep 08 14:08:08.736078 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735776    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb32918-a428-4bc0-a5bb-c22184f68e08" containerName="kube-rbac-proxy"
Sep 08 14:08:08.736078 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735783    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kserve-container"
Sep 08 14:08:08.736078 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.735789    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1655960d-4ef9-47b9-97de-c266d9e1d18e" containerName="kube-rbac-proxy"
Sep 08 14:08:08.738559 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.738541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:08.740678 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.740657    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2qsb2\"/\"kube-root-ca.crt\""
Sep 08 14:08:08.740678 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.740661    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2qsb2\"/\"default-dockercfg-zmrrm\""
Sep 08 14:08:08.741190 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.741174    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2qsb2\"/\"openshift-service-ca.crt\""
Sep 08 14:08:08.749108 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.749090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2qsb2/must-gather-5rt2n"]
Sep 08 14:08:08.758078 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.758059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:08.784373 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.784339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxx22\" (UniqueName: \"kubernetes.io/projected/f47e27ca-f217-443d-bf8a-e466ed8cee36-kube-api-access-hxx22\") pod \"must-gather-5rt2n\" (UID: \"f47e27ca-f217-443d-bf8a-e466ed8cee36\") " pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:08.784543 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.784384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f47e27ca-f217-443d-bf8a-e466ed8cee36-must-gather-output\") pod \"must-gather-5rt2n\" (UID: \"f47e27ca-f217-443d-bf8a-e466ed8cee36\") " pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:08.885097 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.885065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxx22\" (UniqueName: \"kubernetes.io/projected/f47e27ca-f217-443d-bf8a-e466ed8cee36-kube-api-access-hxx22\") pod \"must-gather-5rt2n\" (UID: \"f47e27ca-f217-443d-bf8a-e466ed8cee36\") " pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:08.885097 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.885104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f47e27ca-f217-443d-bf8a-e466ed8cee36-must-gather-output\") pod \"must-gather-5rt2n\" (UID: \"f47e27ca-f217-443d-bf8a-e466ed8cee36\") " pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:08.885474 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.885448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f47e27ca-f217-443d-bf8a-e466ed8cee36-must-gather-output\") pod \"must-gather-5rt2n\" (UID: \"f47e27ca-f217-443d-bf8a-e466ed8cee36\") " pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:08.894213 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:08.894188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxx22\" (UniqueName: \"kubernetes.io/projected/f47e27ca-f217-443d-bf8a-e466ed8cee36-kube-api-access-hxx22\") pod \"must-gather-5rt2n\" (UID: \"f47e27ca-f217-443d-bf8a-e466ed8cee36\") " pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:09.060308 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:09.060219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:09.331585 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:09.331561    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2qsb2/must-gather-5rt2n"]
Sep 08 14:08:09.333720 ip-10-0-140-239 kubenswrapper[2580]: W0908 14:08:09.333689    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf47e27ca_f217_443d_bf8a_e466ed8cee36.slice/crio-5d13e8e871dd983ec6c53e3f3c82f4195b8d5ca850a7f864045af60c8bf37154 WatchSource:0}: Error finding container 5d13e8e871dd983ec6c53e3f3c82f4195b8d5ca850a7f864045af60c8bf37154: Status 404 returned error can't find the container with id 5d13e8e871dd983ec6c53e3f3c82f4195b8d5ca850a7f864045af60c8bf37154
Sep 08 14:08:09.335623 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:09.335600    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 14:08:10.214424 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:10.214369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2qsb2/must-gather-5rt2n" event={"ID":"f47e27ca-f217-443d-bf8a-e466ed8cee36","Type":"ContainerStarted","Data":"5d13e8e871dd983ec6c53e3f3c82f4195b8d5ca850a7f864045af60c8bf37154"}
Sep 08 14:08:10.231849 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:10.231803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:12.088110 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:12.088069    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 08 14:08:15.240821 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.240786    2580 generic.go:356] "Generic (PLEG): container finished" podID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerID="399ef8bda34ac40ea0e1bdb64c1e1fabdaff6fdbdb7a299c1d8c0ed608e3da3f" exitCode=0
Sep 08 14:08:15.241290 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.240832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" event={"ID":"6077a8b1-0ed9-4e61-ba96-09329e8af889","Type":"ContainerDied","Data":"399ef8bda34ac40ea0e1bdb64c1e1fabdaff6fdbdb7a299c1d8c0ed608e3da3f"}
Sep 08 14:08:15.677491 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.677464    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 14:08:15.743626 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.743598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxcdj\" (UniqueName: \"kubernetes.io/projected/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-api-access-lxcdj\") pod \"6077a8b1-0ed9-4e61-ba96-09329e8af889\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") "
Sep 08 14:08:15.743752 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.743728    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6077a8b1-0ed9-4e61-ba96-09329e8af889-proxy-tls\") pod \"6077a8b1-0ed9-4e61-ba96-09329e8af889\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") "
Sep 08 14:08:15.743816 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.743756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-rbac-proxy-sar-config\") pod \"6077a8b1-0ed9-4e61-ba96-09329e8af889\" (UID: \"6077a8b1-0ed9-4e61-ba96-09329e8af889\") "
Sep 08 14:08:15.744237 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.744196    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6077a8b1-0ed9-4e61-ba96-09329e8af889" (UID: "6077a8b1-0ed9-4e61-ba96-09329e8af889"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:08:15.746764 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.746723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6077a8b1-0ed9-4e61-ba96-09329e8af889-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6077a8b1-0ed9-4e61-ba96-09329e8af889" (UID: "6077a8b1-0ed9-4e61-ba96-09329e8af889"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:08:15.746764 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.746730    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-api-access-lxcdj" (OuterVolumeSpecName: "kube-api-access-lxcdj") pod "6077a8b1-0ed9-4e61-ba96-09329e8af889" (UID: "6077a8b1-0ed9-4e61-ba96-09329e8af889"). InnerVolumeSpecName "kube-api-access-lxcdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:08:15.844602 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.844509    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6077a8b1-0ed9-4e61-ba96-09329e8af889-proxy-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 14:08:15.844602 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.844531    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 14:08:15.844602 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:15.844541    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxcdj\" (UniqueName: \"kubernetes.io/projected/6077a8b1-0ed9-4e61-ba96-09329e8af889-kube-api-access-lxcdj\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 14:08:16.247092 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:16.247009    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"
Sep 08 14:08:16.247092 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:16.247022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7" event={"ID":"6077a8b1-0ed9-4e61-ba96-09329e8af889","Type":"ContainerDied","Data":"66bc59d8dc44c2106ddf602904d43b39f84cdec4a2d9e76ad9d7f75dc67eb59c"}
Sep 08 14:08:16.247597 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:16.247110    2580 scope.go:117] "RemoveContainer" containerID="f29748f1ed91757750d2a6e63b947e20a6a37a5b89f57166aee13c1d7bbf3bca"
Sep 08 14:08:16.249210 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:16.249183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2qsb2/must-gather-5rt2n" event={"ID":"f47e27ca-f217-443d-bf8a-e466ed8cee36","Type":"ContainerStarted","Data":"90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650"}
Sep 08 14:08:16.249315 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:16.249225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2qsb2/must-gather-5rt2n" event={"ID":"f47e27ca-f217-443d-bf8a-e466ed8cee36","Type":"ContainerStarted","Data":"49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931"}
Sep 08 14:08:16.256238 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:16.256220    2580 scope.go:117] "RemoveContainer" containerID="399ef8bda34ac40ea0e1bdb64c1e1fabdaff6fdbdb7a299c1d8c0ed608e3da3f"
Sep 08 14:08:16.269912 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:16.269853    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2qsb2/must-gather-5rt2n" podStartSLOduration=1.878897846 podStartE2EDuration="8.269832183s" totalImagesPullingTime="6.390934337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:08:16.266173557 +0000 UTC m=+3108.966104188" watchObservedRunningTime="2026-09-08 14:08:16.269832183 +0000 UTC m=+3108.969762819"
Sep 08 14:08:16.279502 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:16.279391    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"]
Sep 08 14:08:16.285233 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:16.285209    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce1f6-predictor-76644689b9-smjt7"]
Sep 08 14:08:17.750891 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:17.750853    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" path="/var/lib/kubelet/pods/6077a8b1-0ed9-4e61-ba96-09329e8af889/volumes"
Sep 08 14:08:37.339775 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:37.339743    2580 generic.go:356] "Generic (PLEG): container finished" podID="f47e27ca-f217-443d-bf8a-e466ed8cee36" containerID="49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931" exitCode=0
Sep 08 14:08:37.340183 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:37.339824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2qsb2/must-gather-5rt2n" event={"ID":"f47e27ca-f217-443d-bf8a-e466ed8cee36","Type":"ContainerDied","Data":"49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931"}
Sep 08 14:08:37.340183 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:37.340113    2580 scope.go:117] "RemoveContainer" containerID="49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931"
Sep 08 14:08:37.938230 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:37.938195    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2qsb2_must-gather-5rt2n_f47e27ca-f217-443d-bf8a-e466ed8cee36/gather/0.log"
Sep 08 14:08:38.403104 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.403065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:38.549099 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.549072    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p58cr/must-gather-bwz42"]
Sep 08 14:08:38.549603 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.549513    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container"
Sep 08 14:08:38.549603 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.549535    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container"
Sep 08 14:08:38.549603 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.549555    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kube-rbac-proxy"
Sep 08 14:08:38.549603 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.549563    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kube-rbac-proxy"
Sep 08 14:08:38.549865 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.549664    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kube-rbac-proxy"
Sep 08 14:08:38.549865 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.549678    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6077a8b1-0ed9-4e61-ba96-09329e8af889" containerName="kserve-container"
Sep 08 14:08:38.551851 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.551836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p58cr/must-gather-bwz42"
Sep 08 14:08:38.554852 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.554830    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p58cr\"/\"openshift-service-ca.crt\""
Sep 08 14:08:38.554954 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.554885    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p58cr\"/\"kube-root-ca.crt\""
Sep 08 14:08:38.554954 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.554942    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p58cr\"/\"default-dockercfg-zjqml\""
Sep 08 14:08:38.573336 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.573313    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p58cr/must-gather-bwz42"]
Sep 08 14:08:38.574391 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.574372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:38.638810 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.638777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d50aa0bf-ab49-487d-8349-7f8968fd9be3-must-gather-output\") pod \"must-gather-bwz42\" (UID: \"d50aa0bf-ab49-487d-8349-7f8968fd9be3\") " pod="openshift-must-gather-p58cr/must-gather-bwz42"
Sep 08 14:08:38.638810 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.638811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrscs\" (UniqueName: \"kubernetes.io/projected/d50aa0bf-ab49-487d-8349-7f8968fd9be3-kube-api-access-zrscs\") pod \"must-gather-bwz42\" (UID: \"d50aa0bf-ab49-487d-8349-7f8968fd9be3\") " pod="openshift-must-gather-p58cr/must-gather-bwz42"
Sep 08 14:08:38.656501 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.656432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:38.739488 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.739441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d50aa0bf-ab49-487d-8349-7f8968fd9be3-must-gather-output\") pod \"must-gather-bwz42\" (UID: \"d50aa0bf-ab49-487d-8349-7f8968fd9be3\") " pod="openshift-must-gather-p58cr/must-gather-bwz42"
Sep 08 14:08:38.739488 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.739493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrscs\" (UniqueName: \"kubernetes.io/projected/d50aa0bf-ab49-487d-8349-7f8968fd9be3-kube-api-access-zrscs\") pod \"must-gather-bwz42\" (UID: \"d50aa0bf-ab49-487d-8349-7f8968fd9be3\") " pod="openshift-must-gather-p58cr/must-gather-bwz42"
Sep 08 14:08:38.739798 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.739762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d50aa0bf-ab49-487d-8349-7f8968fd9be3-must-gather-output\") pod \"must-gather-bwz42\" (UID: \"d50aa0bf-ab49-487d-8349-7f8968fd9be3\") " pod="openshift-must-gather-p58cr/must-gather-bwz42"
Sep 08 14:08:38.749991 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.749971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrscs\" (UniqueName: \"kubernetes.io/projected/d50aa0bf-ab49-487d-8349-7f8968fd9be3-kube-api-access-zrscs\") pod \"must-gather-bwz42\" (UID: \"d50aa0bf-ab49-487d-8349-7f8968fd9be3\") " pod="openshift-must-gather-p58cr/must-gather-bwz42"
Sep 08 14:08:38.860974 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:38.860935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p58cr/must-gather-bwz42"
Sep 08 14:08:39.127434 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:39.127333    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p58cr/must-gather-bwz42"]
Sep 08 14:08:39.129838 ip-10-0-140-239 kubenswrapper[2580]: W0908 14:08:39.129809    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd50aa0bf_ab49_487d_8349_7f8968fd9be3.slice/crio-ad2d668789037281a1708612f4d41c0b15913dc0cb1687610ab80e0c6748dbf7 WatchSource:0}: Error finding container ad2d668789037281a1708612f4d41c0b15913dc0cb1687610ab80e0c6748dbf7: Status 404 returned error can't find the container with id ad2d668789037281a1708612f4d41c0b15913dc0cb1687610ab80e0c6748dbf7
Sep 08 14:08:39.350218 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:39.350186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p58cr/must-gather-bwz42" event={"ID":"d50aa0bf-ab49-487d-8349-7f8968fd9be3","Type":"ContainerStarted","Data":"ad2d668789037281a1708612f4d41c0b15913dc0cb1687610ab80e0c6748dbf7"}
Sep 08 14:08:42.365153 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:42.365112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p58cr/must-gather-bwz42" event={"ID":"d50aa0bf-ab49-487d-8349-7f8968fd9be3","Type":"ContainerStarted","Data":"0801ad5e6fc374d090382adb2b0c11a13663131887e830dea4785252fa2d3949"}
Sep 08 14:08:42.365618 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:42.365167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p58cr/must-gather-bwz42" event={"ID":"d50aa0bf-ab49-487d-8349-7f8968fd9be3","Type":"ContainerStarted","Data":"4e7e043122cc39894573a0929070765bbe7933bdb9e8f5ca1f2e3df7dbdf410a"}
Sep 08 14:08:42.380948 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:42.380882    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p58cr/must-gather-bwz42" podStartSLOduration=1.633353993 podStartE2EDuration="4.380860766s" totalImagesPullingTime="2.747506773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:08:42.379816283 +0000 UTC m=+3135.079746919" watchObservedRunningTime="2026-09-08 14:08:42.380860766 +0000 UTC m=+3135.080791401"
Sep 08 14:08:43.337684 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.337651    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xjzbp_9377b074-311e-4979-b626-40b1bf8bfccb/global-pull-secret-syncer/0.log"
Sep 08 14:08:43.380325 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.380273    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-2qsb2/must-gather-5rt2n"]
Sep 08 14:08:43.380841 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.380522    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-2qsb2/must-gather-5rt2n" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" containerName="copy" containerID="cri-o://90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650" gracePeriod=2
Sep 08 14:08:43.382816 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.382663    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-2qsb2/must-gather-5rt2n"]
Sep 08 14:08:43.383373 ip-10-0-140-239 kubenswrapper[2580]: E0908 14:08:43.383224    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5rt2n\" is forbidden: User \"system:node:ip-10-0-140-239.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2qsb2\": no relationship found between node 'ip-10-0-140-239.ec2.internal' and this object" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:43.396976 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.396950    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log"
Sep 08 14:08:43.523194 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.523165    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-239.ec2.internal_f2bc47ecdba8650fc42750ab52a3bbbb/haproxy/0.log"
Sep 08 14:08:43.642501 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.642253    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2qsb2_must-gather-5rt2n_f47e27ca-f217-443d-bf8a-e466ed8cee36/copy/0.log"
Sep 08 14:08:43.646134 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.646106    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:43.648271 ip-10-0-140-239 kubenswrapper[2580]: E0908 14:08:43.648236    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5rt2n\" is forbidden: User \"system:node:ip-10-0-140-239.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2qsb2\": no relationship found between node 'ip-10-0-140-239.ec2.internal' and this object" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:43.785927 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.785886    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxx22\" (UniqueName: \"kubernetes.io/projected/f47e27ca-f217-443d-bf8a-e466ed8cee36-kube-api-access-hxx22\") pod \"f47e27ca-f217-443d-bf8a-e466ed8cee36\" (UID: \"f47e27ca-f217-443d-bf8a-e466ed8cee36\") "
Sep 08 14:08:43.786101 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.785984    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f47e27ca-f217-443d-bf8a-e466ed8cee36-must-gather-output\") pod \"f47e27ca-f217-443d-bf8a-e466ed8cee36\" (UID: \"f47e27ca-f217-443d-bf8a-e466ed8cee36\") "
Sep 08 14:08:43.787241 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.787206    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e27ca-f217-443d-bf8a-e466ed8cee36-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f47e27ca-f217-443d-bf8a-e466ed8cee36" (UID: "f47e27ca-f217-443d-bf8a-e466ed8cee36"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:43.788117 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.788087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47e27ca-f217-443d-bf8a-e466ed8cee36-kube-api-access-hxx22" (OuterVolumeSpecName: "kube-api-access-hxx22") pod "f47e27ca-f217-443d-bf8a-e466ed8cee36" (UID: "f47e27ca-f217-443d-bf8a-e466ed8cee36"). InnerVolumeSpecName "kube-api-access-hxx22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:08:43.887389 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.886968    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f47e27ca-f217-443d-bf8a-e466ed8cee36-must-gather-output\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 14:08:43.887389 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:43.887004    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxx22\" (UniqueName: \"kubernetes.io/projected/f47e27ca-f217-443d-bf8a-e466ed8cee36-kube-api-access-hxx22\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\""
Sep 08 14:08:44.375907 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:44.375867    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2qsb2_must-gather-5rt2n_f47e27ca-f217-443d-bf8a-e466ed8cee36/copy/0.log"
Sep 08 14:08:44.376628 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:44.376597    2580 generic.go:356] "Generic (PLEG): container finished" podID="f47e27ca-f217-443d-bf8a-e466ed8cee36" containerID="90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650" exitCode=143
Sep 08 14:08:44.376758 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:44.376702    2580 scope.go:117] "RemoveContainer" containerID="90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650"
Sep 08 14:08:44.376758 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:44.376746    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2qsb2/must-gather-5rt2n"
Sep 08 14:08:44.392575 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:44.392552    2580 scope.go:117] "RemoveContainer" containerID="49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931"
Sep 08 14:08:44.406988 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:44.406967    2580 scope.go:117] "RemoveContainer" containerID="90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650"
Sep 08 14:08:44.407297 ip-10-0-140-239 kubenswrapper[2580]: E0908 14:08:44.407270    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650\": container with ID starting with 90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650 not found: ID does not exist" containerID="90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650"
Sep 08 14:08:44.407341 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:44.407308    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650"} err="failed to get container status \"90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650\": rpc error: code = NotFound desc = could not find container \"90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650\": container with ID starting with 90b331a7a97dd7ba21ac63c17d5b9deb0fe7cb85a2d5290b8b4d0fffd4932650 not found: ID does not exist"
Sep 08 14:08:44.407341 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:44.407327    2580 scope.go:117] "RemoveContainer" containerID="49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931"
Sep 08 14:08:44.407664 ip-10-0-140-239 kubenswrapper[2580]: E0908 14:08:44.407642    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931\": container with ID starting with 49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931 not found: ID does not exist" containerID="49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931"
Sep 08 14:08:44.407755 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:44.407677    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931"} err="failed to get container status \"49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931\": rpc error: code = NotFound desc = could not find container \"49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931\": container with ID starting with 49364900040dc21f3c84efdae481061a2a6f06615bc99158e6ffcb5f05b82931 not found: ID does not exist"
Sep 08 14:08:45.753265 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:45.753224    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" path="/var/lib/kubelet/pods/f47e27ca-f217-443d-bf8a-e466ed8cee36/volumes"
Sep 08 14:08:47.915206 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:47.915173    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-tjj2c_4cb6c622-93eb-4efd-b040-d7ba5894398b/cluster-monitoring-operator/0.log"
Sep 08 14:08:47.950588 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:47.950552    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-s97st_8b41c3aa-e580-463a-a272-1facc973a66a/kube-state-metrics/0.log"
Sep 08 14:08:48.014383 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.014266    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-s97st_8b41c3aa-e580-463a-a272-1facc973a66a/kube-rbac-proxy-main/0.log"
Sep 08 14:08:48.064050 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.063988    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-s97st_8b41c3aa-e580-463a-a272-1facc973a66a/kube-rbac-proxy-self/0.log"
Sep 08 14:08:48.124913 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.124867    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d9bd9bbd9-rkz2x_649d85bf-146e-4b48-a20c-82e20d920117/metrics-server/0.log"
Sep 08 14:08:48.125703 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.125612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:48.194388 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.194341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d6qxw_f31cde05-3241-42b0-bf2a-5db1642aab89/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:48.204259 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.204222    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8q6mk_7896a071-d3fe-4acb-837d-bf0be2356915/node-exporter/0.log"
Sep 08 14:08:48.244973 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.244948    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8q6mk_7896a071-d3fe-4acb-837d-bf0be2356915/kube-rbac-proxy/0.log"
Sep 08 14:08:48.295645 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.295569    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8q6mk_7896a071-d3fe-4acb-837d-bf0be2356915/init-textfile/0.log"
Sep 08 14:08:48.532219 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.532184    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-bdc9h_fb556705-9f11-4fe6-aa5a-ad7fd14743ec/kube-rbac-proxy-main/0.log"
Sep 08 14:08:48.556339 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.556239    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-bdc9h_fb556705-9f11-4fe6-aa5a-ad7fd14743ec/kube-rbac-proxy-self/0.log"
Sep 08 14:08:48.582487 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.582439    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-bdc9h_fb556705-9f11-4fe6-aa5a-ad7fd14743ec/openshift-state-metrics/0.log"
Sep 08 14:08:48.634122 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.634083    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78ad7b9c-2f97-431b-b270-554d8e1e2913/prometheus/0.log"
Sep 08 14:08:48.661498 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.661457    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78ad7b9c-2f97-431b-b270-554d8e1e2913/config-reloader/0.log"
Sep 08 14:08:48.693799 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.693768    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78ad7b9c-2f97-431b-b270-554d8e1e2913/thanos-sidecar/0.log"
Sep 08 14:08:48.730326 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.730292    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78ad7b9c-2f97-431b-b270-554d8e1e2913/kube-rbac-proxy-web/0.log"
Sep 08 14:08:48.762084 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.762050    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78ad7b9c-2f97-431b-b270-554d8e1e2913/kube-rbac-proxy/0.log"
Sep 08 14:08:48.794229 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.794198    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78ad7b9c-2f97-431b-b270-554d8e1e2913/kube-rbac-proxy-thanos/0.log"
Sep 08 14:08:48.825536 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.825513    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78ad7b9c-2f97-431b-b270-554d8e1e2913/init-config-reloader/0.log"
Sep 08 14:08:48.926032 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:48.926002    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-k7rsm_8c0a4ef9-eef8-4730-a6c4-ed33c7079eab/prometheus-operator-admission-webhook/0.log"
Sep 08 14:08:51.698275 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.698224    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"]
Sep 08 14:08:51.698884 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.698731    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" containerName="copy"
Sep 08 14:08:51.698884 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.698752    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" containerName="copy"
Sep 08 14:08:51.698884 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.698769    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" containerName="gather"
Sep 08 14:08:51.698884 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.698778    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" containerName="gather"
Sep 08 14:08:51.698884 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.698881    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" containerName="copy"
Sep 08 14:08:51.699153 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.698895    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47e27ca-f217-443d-bf8a-e466ed8cee36" containerName="gather"
Sep 08 14:08:51.702612 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.702593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.716805 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.716738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"]
Sep 08 14:08:51.875173 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.875135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-sys\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.875372 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.875261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-proc\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.875372 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.875317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s5pm\" (UniqueName: \"kubernetes.io/projected/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-kube-api-access-5s5pm\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.875523 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.875377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-lib-modules\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.875827 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.875803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-podres\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.977485 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.977376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-sys\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.977485 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.977460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-proc\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.977485 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.977483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s5pm\" (UniqueName: \"kubernetes.io/projected/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-kube-api-access-5s5pm\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.977718 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.977497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-sys\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.977718 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.977509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-lib-modules\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.977718 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.977556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-podres\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.977718 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.977580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-proc\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.977718 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.977689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-podres\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.977884 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.977733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-lib-modules\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:51.987773 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:51.987741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s5pm\" (UniqueName: \"kubernetes.io/projected/5ffa9469-989e-48f1-bb94-d38cc7ccb84d-kube-api-access-5s5pm\") pod \"perf-node-gather-daemonset-fcwhd\" (UID: \"5ffa9469-989e-48f1-bb94-d38cc7ccb84d\") " pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:52.013545 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:52.013507    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:52.310482 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:52.310454    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"]
Sep 08 14:08:52.313061 ip-10-0-140-239 kubenswrapper[2580]: W0908 14:08:52.313029    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5ffa9469_989e_48f1_bb94_d38cc7ccb84d.slice/crio-dc9d85aac029adaed6a988298d992a9d56cb32b9df7fb783819ef41f0d1f9f52 WatchSource:0}: Error finding container dc9d85aac029adaed6a988298d992a9d56cb32b9df7fb783819ef41f0d1f9f52: Status 404 returned error can't find the container with id dc9d85aac029adaed6a988298d992a9d56cb32b9df7fb783819ef41f0d1f9f52
Sep 08 14:08:52.434320 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:52.434285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd" event={"ID":"5ffa9469-989e-48f1-bb94-d38cc7ccb84d","Type":"ContainerStarted","Data":"563b18e9ab1027d8d89978cd3a36d910feac5add8e3801013fbf0fe16785b248"}
Sep 08 14:08:52.434555 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:52.434351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd" event={"ID":"5ffa9469-989e-48f1-bb94-d38cc7ccb84d","Type":"ContainerStarted","Data":"dc9d85aac029adaed6a988298d992a9d56cb32b9df7fb783819ef41f0d1f9f52"}
Sep 08 14:08:52.434555 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:52.434492    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:08:52.453367 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:52.452478    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd" podStartSLOduration=1.452459707 podStartE2EDuration="1.452459707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:08:52.450530158 +0000 UTC m=+3145.150460797" watchObservedRunningTime="2026-09-08 14:08:52.452459707 +0000 UTC m=+3145.152390341"
Sep 08 14:08:53.132933 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:53.132896    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8mnkp_d011fe55-25c5-486c-850f-49d3abbd6650/dns/0.log"
Sep 08 14:08:53.157573 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:53.157544    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8mnkp_d011fe55-25c5-486c-850f-49d3abbd6650/kube-rbac-proxy/0.log"
Sep 08 14:08:53.295828 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:53.295798    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-khkzf_c7c65dcc-4c14-4d6d-a71f-07f1ff2e4862/dns-node-resolver/0.log"
Sep 08 14:08:53.811950 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:53.811923    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4m5ck_aceec167-d19a-4695-8412-94b4c07bea07/node-ca/0.log"
Sep 08 14:08:55.021653 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:55.021625    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tzvsh_e8475a96-b21d-4bd9-9bd7-c1bebed877d2/serve-healthcheck-canary/0.log"
Sep 08 14:08:55.465889 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:55.465865    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bdzv9_269833ab-86c1-409b-957b-9bd903c7ea86/kube-rbac-proxy/0.log"
Sep 08 14:08:55.487996 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:55.487967    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bdzv9_269833ab-86c1-409b-957b-9bd903c7ea86/exporter/0.log"
Sep 08 14:08:55.510144 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:55.510115    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bdzv9_269833ab-86c1-409b-957b-9bd903c7ea86/extractor/0.log"
Sep 08 14:08:58.126871 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:58.126834    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nt9s7_fe3d1256-34af-4575-b90a-1b0bd45e3707/s3-init/0.log"
Sep 08 14:08:58.144415 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:58.144380    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nt9s7_fe3d1256-34af-4575-b90a-1b0bd45e3707/download-hf-model/0.log"
Sep 08 14:08:58.449808 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:08:58.449728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p58cr/perf-node-gather-daemonset-fcwhd"
Sep 08 14:09:02.475740 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:02.475701    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fqz9r_0d140ce0-20e9-4607-ad37-8c741949a264/migrator/0.log"
Sep 08 14:09:02.498847 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:02.498770    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fqz9r_0d140ce0-20e9-4607-ad37-8c741949a264/graceful-termination/0.log"
Sep 08 14:09:02.870215 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:02.870183    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-jt8hk_7994c758-9e77-4100-9449-c944e87bf430/kube-storage-version-migrator-operator/1.log"
Sep 08 14:09:02.871316 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:02.871286    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-jt8hk_7994c758-9e77-4100-9449-c944e87bf430/kube-storage-version-migrator-operator/0.log"
Sep 08 14:09:04.087685 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.087655    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n86t6_824e0f88-9b13-4a1d-bf08-044dcb419c43/kube-multus-additional-cni-plugins/0.log"
Sep 08 14:09:04.111349 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.111308    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n86t6_824e0f88-9b13-4a1d-bf08-044dcb419c43/egress-router-binary-copy/0.log"
Sep 08 14:09:04.132316 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.132290    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n86t6_824e0f88-9b13-4a1d-bf08-044dcb419c43/cni-plugins/0.log"
Sep 08 14:09:04.158177 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.158147    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n86t6_824e0f88-9b13-4a1d-bf08-044dcb419c43/bond-cni-plugin/0.log"
Sep 08 14:09:04.182861 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.182836    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n86t6_824e0f88-9b13-4a1d-bf08-044dcb419c43/routeoverride-cni/0.log"
Sep 08 14:09:04.204947 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.204920    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n86t6_824e0f88-9b13-4a1d-bf08-044dcb419c43/whereabouts-cni-bincopy/0.log"
Sep 08 14:09:04.227223 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.227190    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n86t6_824e0f88-9b13-4a1d-bf08-044dcb419c43/whereabouts-cni/0.log"
Sep 08 14:09:04.429832 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.429760    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tswnj_496cfe43-b060-47e7-a81b-7648af1924f2/kube-multus/0.log"
Sep 08 14:09:04.530889 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.530849    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dbk62_c009591a-887d-4708-a981-bc6005a5a690/network-metrics-daemon/0.log"
Sep 08 14:09:04.566010 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:04.565978    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dbk62_c009591a-887d-4708-a981-bc6005a5a690/kube-rbac-proxy/0.log"
Sep 08 14:09:05.393927 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:05.393891    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd6fv_e8a88575-8309-4038-9720-4bc5e427d79c/ovn-controller/0.log"
Sep 08 14:09:05.413256 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:05.413226    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd6fv_e8a88575-8309-4038-9720-4bc5e427d79c/ovn-acl-logging/0.log"
Sep 08 14:09:05.435252 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:05.435223    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd6fv_e8a88575-8309-4038-9720-4bc5e427d79c/kube-rbac-proxy-node/0.log"
Sep 08 14:09:05.456776 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:05.456748    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd6fv_e8a88575-8309-4038-9720-4bc5e427d79c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 08 14:09:05.476509 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:05.476480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd6fv_e8a88575-8309-4038-9720-4bc5e427d79c/northd/0.log"
Sep 08 14:09:05.496587 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:05.496559    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd6fv_e8a88575-8309-4038-9720-4bc5e427d79c/nbdb/0.log"
Sep 08 14:09:05.527934 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:05.527908    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd6fv_e8a88575-8309-4038-9720-4bc5e427d79c/sbdb/0.log"
Sep 08 14:09:05.567857 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:05.567813    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd6fv_e8a88575-8309-4038-9720-4bc5e427d79c/ovnkube-controller/0.log"
Sep 08 14:09:07.040427 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:07.040379    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-b465g_704bbd00-297b-4981-ad29-944ecf46aaf0/check-endpoints/0.log"
Sep 08 14:09:07.068716 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:07.068674    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5xdww_708bf191-c98f-4529-be49-f8914698261a/network-check-target-container/0.log"
Sep 08 14:09:08.057107 ip-10-0-140-239 kubenswrapper[2580]: I0908 14:09:08.057079    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5rfhl_b8b42de1-4263-4bd4-be8e-8b2e65f9da13/iptables-alerter/0.log"